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I. SUMMARY 



The ERIC Clearinghouse on Educational Media and Technology was asked 
by the U.S. Office of Education to help evaluate on-line computer searching 
of the ERIC document file. A terminal with Its supporting equipment was 
Installed In the clearinghouse, and arrangements were made for data-phone 
linkage with Lockheed's nearby Information Sciences facilities, where the 
DIALOG system was available. 

DIALOG was to be used to accomplish clearinghouse tasks — both those 
usually performed and those Impossible without such a system — and to see 
If Individuals could sit down at a terminal and, with little preliminary 
Instruction, use such a system to locate relevant educational research 
dociiments. The system was employed In a variety of clearinghouse projects 
and proved extremely useful, but the focus of this report Is on nine case 
studies. These reveal In some detail the reactions of the Individuals — 
who had quite different backgrounds and sought quite different Information — 
to two-hour hands-on sessions at the terminal. 

The case studies demonstrate that Individuals with no preliminary 
experience with computers or mechanized Information systems can Indeed, 

In a reasonably short time, use a system such as DIALOG effectively. 

The two aspects of the system which most Impressed the nine evaluators 
were Its speed and the way It "widened horizons" — the way It suggested 
other relevant areas of Information or different approaches to the Infor- 
mation originally sought. Verbatim comments from the nine evaluators are 
presented throughout the report. 
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When the evaluators were specifically asked to list good features of 
the system. Its speed and Its "horizon-widening" effect were mentioned. 

The way that sets of documents could be combined, using Boolean logic, 
and the fact that the system was simple to use also were cited. 

The evaluators were specifically asked to list bad features, &nd delays 
In waiting for the system to execute a command were singled out, along 
with the feeling that considerable experience or time was needed to master 
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the operating rules. (While some evaluators made this last comment, 
different ones called the system simple to use and easy to work with.) 

Users were asked to comment on each component of the system, such as the 
typed terminal record and the cathode ray tube display. The terminal 
record was apprecl;a'ted~ as a means of monitoring the ongoing search, and 
as a way of obtaining a permanent record of search strategy. The cathode 
ray tube was cited as saving great amounts of time and being a natural 
way to present data. There was criticism of some components, or of their 
Integration In the system. A number of users wished commands could be 
entered without having to hit so many keys. One section of the r&^>ort 
lists changes which have been made In DIALOG since the evaluation sessions; 
some of the unfavorable comments are obviated by the changes. 

Generally favorable reactions to the system from clearinghouse 
personnel are reported In the last section of the report, written by the 
project coordinator who spent more than 120 hours at the terminal herself. 
Concern Is registered over problems with both hardware and software, which 
caused Interruptions In scheduled service from time to time. Many of these 
problems were the natural result of Lockheed continuing to develop the 
system while It was In use. 
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To assist in explaining the DIALOG system, a set of 27 slides was 
prepared to demonstrate a fairly Involved search for ethnic studies materials. 
A copy of the script for the slide presentation Is available free from the 
clearinghouse, and a set of the slides can be purchased from the 
Information Research Projects Fund (Attn. C. Collins, ERIC at Stanford) 

for $10. 

Finally, the cooperation, assistance and good will of the Lockheed 
Information Sciences Group—and especially Roger Summit and Don Schaaf— 
are acknowledged with appreciation, as Is the aid and counsel of James J. 
Prevel of the U.S. Office of Education. 
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II. INTRODUCTION 



A. Why the Study Was Conducted 



Because of staffmembers* Interest In Information retrieval techniques 
and because of Its proximity to the Lockheed Information Sciences Laboratory » 
the ERIC Clearinghouse at Stanford University was asked to help evaluate 
on-line. Interactive, searching of the 12,300 document ERIC file. The 



Lockheed DIALOG system was considered particularly suited to the analysis 
of user performance and satisfaction because It turns out a printed record 
of each step In search procedures. Because DIALOG was developed and was 
being maintained so close to the Stanford campus. It was relatively simple 
to make It available In the clearinghouse. A two-piece remote access 
terminal (printer unit and keyboard/cathode ray tube display unit) had to 
be Installed, along with Interfacing equipment, but line charges for 
connection with the Lockheed computer were relatively low because of the 
short distance Involved. Other steps Included obtaining the ERIC file on 
magnetic tape and converting It for data cell access. 

Initial plans were to devote 75% of the slx-hours-per-week system 
time to regular clearinghouse use, and to Invite a variety of users to 
pursue their educational Interests during the other 25% of the time. 

Over the months the actual split of time came close to the planned division, 
but a good many more people were exposed to DIALOG than that suggests: 



Regular clearinghouse activity Includes answering user requests for a 
variety of Information, and this form of user service was greatly 
facilitated by having DIALOG available. DIALOG-produced prlnt-outs were 
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mailed in response to a number of search requests, and many users visited 
the clearinghouse to observe their searches and make modifying suggestions 
while they were in process. This kind of DIALOG utilization was separate 
and distinct from the two-hour, hands-on evaluating sessions which were 
the basis for the nine case studies. In addition to allowing investigation 
of the potential of on-line retrieval systems, having the terminal Installed 
at the Stanford clearinghouse brought the Office of Education's Interest 
in employing new technology to the attention of influential people. 

The title of this report was chosen to emphasize that the project 
was not designed as an experiment, in the sense of attributing specific 
effects to specific causes. Of necessity we are reporting case studies. 

One of the attractive features of DIALOG is that the strategy employed in 
each search is permanently recorded, along with the responses of the system. 



But even this did not allow objective investigation of what search strategies 
different individuals would favor. 

Two hours is enough time for the average user to become quite capable 
of using the DIALOG system, as this report will show. Two hours is not 
long enough, however, for a user to gain thorough familiarity with all 
the different avenues of inquiry possible with the system. (This is not 
a criticism of DIALOG; with two minutes of instruction the average driver 
could use a Formula Two racing car to go to the corner grocery, though a 
skilled driver could benefit from months of experience before going on 
the circuit with it. It is an admirable design precisely because it can 
be used under such different circumstances and at different levels of 
complexity.) But because it takes longer than two hours to acquire a 















complete familiarity with DIALOG, the search strategy which each user 
employed was almost of necessity the one he was first shown. 
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To determine the search strategies reinforced by DIALOG or those 
favored when all are familiar to users, it would be necessary to make the 
system available to the same group of people over a long period of time. 
Though the sample size is extremely limited (n ■ 1), some evidence on long 
exposure to the system was collected. One of the authors of this report 
scheduled all utilization of DIALOG, introduced users to it, and used 
it herself approximately 120 hours. Her reflections on these experiences 
are presented as section 5 of this report. 

All of the nine evaluators were busy people, and most would not have 
participated in the study without the incentive of being able to pursue 
personally desired information. This precluded any simple experimental 

i 

\. 

design in which different people would be directed to search for the same 
information. The most flagrant weakness in role-playing thus was avoided, 
and the satisfaction with the system felt by most users was satisfaction 
with obtaining meaningful personal assistance. 

This approach was felt to be in perfect accord with the point of view 



which had been expressed by James J. Prevel, Acting Chief of the Bureau 
of Research's Equipment Development Branch: 

The user of the information retrieval system is the ultimate 
judge of which information provided him is relevant to questions 
that he wishes to have answered, REGARDLESS of how he has 
verbalized these to the system. Only the user's paradigms are 
to be served by a system rather than some consensus by others 
who may feel they know what is really wanted by the user, or who 
claim to know what he should want. In order to measure the 
effectiveness of an information retrieval system in providing 
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with obtaining meaningful personal assistance. 
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relevant' Information to users » the questions posed to the system 
must be derived from the real needs of users !i^o are thus 
motivated In some real way to have valid responses. 

In the final analysis . . . , the question to be answered 
Is not "Is the system operating efficiently?" but "Is the system 
worthwhile?" 

We did give visitors the opportunity to pursue Information of vital 

Interest to themselves » but It would be foolish to pretend that there was 

« 

not any role playing; each visitor was playing the role of "Invited guest 
at the clearinghouse" as well as the role of "Individual being confronted 
with new example of technology." (To put this another way» there was lots 
of Hawthorne effect.) The extensive Interview after the DIALOG session 
would. It was hoped, get beyond the role playing, to discover more than 
whether the user was prejudiced toward or against computers or the ERIC 
system. 

B. A Description of the DIALOG System 

Those readere familiar with the eyetem may wish 
to skip thie eeotion of the report ae well ae the 
next^ which preeente a acanple eearoh of the ERIC 
document file. 

The description of DIALOG presented here Is taken, with his permission, 
from a paper previously prepared by the. developer of the system. 
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Roger K. Summit < 



A recent breakthrough In Information retrieval has been the 
use of an on-line computer to assist an Individual In Identifying 
which documents In a large collection pertain to a particular 













Interest topic. By ellmlnstlng Intermediaries and placing the 
user In direct communication with the computer, results are 
available In seconds or minutes Instead of days or weeks, and 
Interesting avenues of exploration suggested by preliminary 
results can be pursued without time lapse • • • • 



Not long ago Information retrieval was a well-understood 
process. One simply went to the library, examined the card 
catalog for an appropriate subject heading, and browsed through 
the catalog card entries under that subject heading. The 
niimbers of the Interesting Items were copied down and the 
documents located and examined. When the collection becomes 
large and diverse, however, much of the Individual's time Is 
spent In data processing; l.e., sorting through catalog cards 
to Identify those of Interest. 



The advent of the computer led to the proposal of several 
automatic searching techniques, most of which Interposed a 
systems expert or Interpreter between the searcher and the store 
of Information. It then became necessary for the prospective 
user to describe his Interest to the systems expert who 
Interpreted the request and translated It Into computer-legible 
form. To gain efficiency, several such search requests were 
"batched," keypunched, and fed Into the computer. The results 
were then separated by request, and reviewed by the systems 
expert before they were returned to the user. It was frequently 
necessary to Input several queries for the same search topic to 
assure coverage of the collection. Because of the physical 
distance between the user and the computer, results were delayed; 
because of the communication distance, results were frequently 
low fidelity In terms of reliance. 

The present generation of on-line computers, which allow 
the searcher to be In direct contact with the computer, suggest 
the possibility of regaining the search flexibility of the manual 
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system In terms of user-corpus proximity, while at the same time 
providing the user the data processing power of a computer* 

With this broad objective, design of the DIALOG system was 
begun. Completed at the end of 1966, the DIALOG language was 
Implemented on an IBM 360/30 computer [and later on a 360/40] • 

• • • • The significance of a computer-oriented coordinate search 
' system Is that It allows Items to be Indexed with very specific 
Index terms (as opposed to broad subject headings), and allows 
the user effectively to build his own personalized subject 
heading at search time by specifying desired and, or, and not 
relationships between Index terms. The final search expression 
for the topic "welding defects In aluminum" could have been: 

"welding" or "arc welding" not 
"pressure welding" 

and 

"defect" or "flaw" or "porosity" 

and 

"aluminum" or "aluminum alloy" 

In this expression there are three concepts, each of which Is 
expressed by multiple terms* 

In any coordinate search there are four phases: 

Identifying and selecting Index terms 

Combining or relating Index terms by and , or, and 
not to define a search expression 

Generating a subfile of Items which meet the 
criteria of the search expression 

Examination of results and modification of search 
expression 

DIALOG commands provide the user a language with which he can 
direct the computer to assist him In accomplishing these four 
phases * 
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Fig. 1--The Terminal, with CRT Display at Right, Printer at Left 




B-:,: 



> REF 


DESCRIPTOR 


citations 


REL. TERMS 


REF 


'v -'it . ^ 


IT/CURRICULUM design 


32 


5 


£1 


E2 


it/curriculum development 


618 


21 


E2 


E3 


IT/CURRICULUM ENRICHMENT 


58 


3 


E3 


E4 


IT/CURRICULUM EVALUATION 


121 


6 


E4 


IS 


*IT/CURRICULUM GUIDES 


415 


14 


ES 


If 


IT/CURRICULUM PLANNING 


175 


9 

4 


Efi 


E7 


IT/CURRICULUM PROBLEMS 


24 


E7 




it/curriculum research 


122 


8 


b W 

E8 


• Ef IT/CURRICULUM RESTRUCTURING PROJECT 

V, EII^TERNEXTCCNMMAND^ 


1 


blP 

E9 






Fig* 2--The CRT Display Screen 
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With the DIALOG retrieval language, the user can browse 
through thousands of citations In seconds. This Is accomplished 
by storing the citations In a remotely located computer, and 
providing the user an Input/output terminal connected to the 
computer over a telephone line. By Issuing commands to the 
computer, the user causes the computer to process Its store of 
Information In much the same basic manner as does the person 
conducting a manual search on a traditional library card catalog. 

The Input/output terminal consists of a keyboard/display 
device (on the right In Flg.l) and console-printer (on the left). 
The keyboard of the display device Is used to communicate 
commands to the computer, whereas the display screen and the 
console-printer are used by the computer to return requested 
Information to the user. As Information Is typed In on the 
display keyboard. It appears on the screen, and errors can be 
corrected by backspacing and overtyping. 

[The DIALOG commands] are actually labels associated with 
the top row of keys on the display keyboard, and are Issued to 
the computer by merely depressing the appropriate key. [Use of 
these commands Is Illustrated In the next section of this report 
with an actual search.] .... 

The DIALOG language offers a powerful, user-directed retrieval 
language which: 

Provides the user a variety of "command" functions 
for communication, search, and display of Information 

Provides flexibility to Include additional commands 
or other operational modes as new search techniques 
are developed 

Assists the user In search definition and In full 
employment of system capabilities 

Allows Intermediate user evaluation of search results 
with subsequent request refinement or modlflcaci{>n 

Accepts any arbitrarily complex or detailed search 
descriptions 






operates effectively with existing Information 
stores without requiring the re-lndexlng of the 
stores 

Requires a minimum of bookkeeping or remembering 
on the part of the user 

Minimizes elapsed time between query and response 

Eliminates need for "middle-man" request 
Interpretation by system specialists 

Allows real-time Interaction between user and 
system for search guidance 

Is easily learned by nonspeclallsts 

'——By Dr. Roger K. Summit 

Information Sciences Laboratory 
Lockheed Missiles and Space Co. 
Palo Alto, California 



C. How Would a DIALOG Search Proceed? 

The commands which allow the searcher to manipulate the ERIC file on 
the computer are quite simple. Each of the special characters on the 
keyboard above the numerals (such as @ and X) stands for one command or 
one type of manipulation. The four principal commands used In searching 
are the EXPAND, SELECT, COMBINE, and DISPLAY. 

Briefly, the EXPAND command can be used to show the searcher two kinds 
of things. First It brings onto the screen a "window" or a "page" of the 
alphabetical Index where a particular term Is located, giving the number 
of citations posted to the terms as well as the number of cross-referenced 
thesaurus terms listed for each (see Fig. 2). Each visible entry Is marked 
with a reference nund>er plus the letter "E" (El, E8, etc.). Secondly the 
EXPAND command brings the cross-referenced entries for a particular term 
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Fig. 3- -A Sample of the Typed Terminal Record 
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Fig, 4--The CRT Display Screen Presenting a Citation 
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onto the screen, again showing the number of posted citations and the 
number of thesaurus entries for each one. In order to Identify which Index 
the term could be found In (author, title, descriptor, etc.) the searcher 
prefixes his term with a two-letter abbreviation Identifying the Index. 

(For example, IT/ Is the prefix for the subject or descriptor categories.) 

The SELECT command allows the searcher to set aside for future use 
any terms— and therefore the documents Indexed by them— which he wants to 
Incorporate In his search. The typed terminal record, which Is produced 
on the printer that Is part of the terminal, records each command and 
Indicates the Identification or set number which Is assigned to each group 
of documents as It Is set aside (see Fig. 3). 

After selecting out every term In the file which relates to each 
concept In his search, the searcher then COMBINES the terms appropriate to 
each concept by adding (In Boolean logic, ORlng) the terms together to 
create a new set. When the selected terms have been grouped according to 
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concept, the concept sets are then COMBINED again, this time so that the 
sets Intersect (In Boolean logic, the terms are ANDed). This second 
combination narrows In on the few documents which are Indexed by at least 



one term from eaoh of the concept sets. 

The resumes for the new, narrowed set of documents can then be 
brought to the screen one by one, using the DISPLAY command (see Fig. A). 
The searcher may now go through what he has retrieved, selecting those 
dociiments which he will wish to examine further In hard copy or microfiche 
form. Finally, the results can be printed off-line using PRINT (this Is 
quite fast), or they can be typed, using the TYPE command, on the terminal 
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printer (this Is much slower). The format for this printout Is also at 
the option of the searcher. 

In the latter part of this section of the report, the search of one 
of the evaluators has been annotated with a running commentary on the 
operations and strategy. The appendix contains the search records of the 
other eight evaluators, one of which Is also annotated. 



As an Introduction to the following search record. It should be said 
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that the evaluator wished to obtain Information on flexible scheduling. 
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SEARCH TITLE: 



date: 



REQUESTOR: 



FLEXIBLE SCHEDULING 

07/23/69 

PERSON H - 2 



SET NO. IN 

COMMAND-OPERAND (S) NO. SET 



The 4 column headings indicate what command was 
given^ what set number was assigned (if a set was 
created) y how many documents are contained in that 

DESCRIPTIOH OF SET 

(■t-OR,*-AHD,-NOT) contents. 
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The atphdbetioal index around the desoriptor^ 
FLEXIBLE SCHEDULINGj Was expanded (E) , 



I 



S-E4 

S-E8 

S-E5 



1 

2 

3 



11 

1 

21 



IT/FLEXIBLE SCHEDULE 
IT/FLINT YOUTH STUDY 
IT/FLEXIBLE SCHEDULING 



There were 11 documents on 
FLEXIBLE SCHEDULE which were 
selected (S) to form set 1, and 
21 documents on FLEXIBLE SCHED- 
ULING which were selected to form 
set 3, Set 2 contains a document 
on the FLINT YOUTH STUDY which 
also appealed in the alphabetical 
display and was of interest to 
the evaluator. 
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E-S^ in order to display its 








thesaurus entries. 
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or addition of sets 1 through 10. 
This set then included all terms 
relating to the concept or idea 
of creative or differing schedul- 
ing methods. 



E-IT/PROGRAMS 



The evaluator now turned to the program aspect 
or concept in her search^ and expanded the index 
around the term FROGRAMS. 



S-E5 

E-E5 



12 50 IT/PROGRAMS 



This was selected to form set 12 ^ and then 
expanded by its reference number to see 
the thesaurus entries. 
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S-E34 


13 


68 


S-E35 


14 


33 


S-E39 


15 


304 


S-E46 


16 


246 


S-E49 


17 


26 


S-E55 


18 


61 


S-E58 


19 


87 


S-E77 


20 


18 


E-IT/PROJECTS 


S-E5 


21 


37 


E-E5 




- 



C-12-21/+ 


22 


868 


0-11*22 


23 


39 



D-23 

K-23/1 

K-23/3 

K-23/5 

K-23/6 

K-23/7 

K-23/8 

K-23/10 

K-23/12 

K-23/13 

K-23/14 

K-23/16 

K-23/17 

K-23/19 

K-23/21 

K-23/22 

K-23/24 

K-23/25 

Yir-23ll(i 

K-23/27-29 



IT/DEMONSmilON PROGRAMS 
IT/DEVELOFMEMTAL PROGRAMS 
IT/EDUCATIONAL PROGRAMS 
IT/EXPERIMEMTAL PROGRAMS 
XT/FOUMDATION PROGRAMS 
IT/INDIVIDUALIZED PROGRAMS 
IT/INSTRUCTIOMAL PROGRAMS 
IT/SCHOOL COMMUNITY PROGRAMS 

The term PROJECTS wae expanded 
IT/PROJECTS and aeleeted to form set 21. 

However^ when it was expanded by 
its reference number^ nothing of 
interest was discovered among 
the thesaurus terms listed under 
it. 

12+13+14+15+16+17+18+19+20f21 The searcher now added 

together the program 
terms to form set 22. 

(l+2+3+4+5+6+7+8+9+10)*(12+13+l4f 15+16+17+18+19+20+21) 
Sets 11 (the scheduling set) and 22 (the program 
or prof eat set) were next combined (C) to form 
an intersection^ the AND operation in Boolean 
logic^ with the resulting set 23 containing 39 
items indexed by at least one term from each 
concept set. 

The resumes for this set were then displayed (D) on 
the screen^ and the relevant ones set aside using 
the keep (K) command, \fhen set aside in this 
manner the items are filed in a reference set 
which is arbitrarily numbered 99. 



Eight more relevant 
terms were located and 
selected to form sets 
13 through 20. 



1-21 ITEMS HAVE BEEN PRINTED 



Finally^ the items put aside in set 99 were 
printed (P)J Format 5 which contains the 
cataloging^ indexing and abstract for each 
document was chosen. After the print had been 
completed off-line at Lockheed^ the results 
were sent to the clearinghouse for forwarding 
to the evaluator. 
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III. HOW THIS STUDY WAS CONDUCTED 



An attempt was made to get people in a variety of professional roles 
to sit down at the remote access terminal for two—hour sessions. The nine 



evaluators are described below » with Identifying letters which are used 
all the way through this report to Indicate which search records and which 
verbatim answers are from which user. 



Professional Description, 
AgS And Sex 



Information Sought, In the 
Visitor *8 Own Words 



Person A 



Researcher engaged In the plan- 
ning, technical aspects and 
conduct of educational research 
projects (28, M). 



Adult basic education programs for 
the disadvantaged unemployed, 
especially as pertains to Industrial 
programs . 



Person B 



Motivational educator In private 
practice, working with children 
referred by schools, doctors, 
etc. (Al, F). 



Instructional Innovation, multi- 
media Instruction and research 
design. 



Person C 



Graduate student In education 
who will return to district to 
work (25, M) . 



Information relating to multimedia 
techniques Involving Instructional 
television, multi-level reading 
materials, simulation games, role 
playing, and 8mm film loops. Also 
materials on conservation projects 
In film and television and attitude 
evaluation techniques for elementary 
school students. 



Person D 
Person E 



M.D. engaged In psychiatric 
research and therapy (29, M) . 



Studies of the perceived environ- 
ment of educational settings. 



Assistant Professor of 
Linguistics and Computer 
Science (27, M) . 



1) Computer science articles for 
Unlv. of San Francisco Computer 
Dept. Information retrieval. 

2) Linguistics, more specifically, 
computational linguistics (my field) . 
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Person F 



Person G 



University librarian directing 
library automation, and Involved 
In developing a different 
on-line retrieval system (40, M). 



Professor of education teaching 
and doing research In educational 
psychology (52, M) . 



Library automation, data manage- 
ment, file organization, library 
technical processing, mlcropubllca- 
tlon (plus many subjects with prefix 
"micro-”) , computer applications to 
text processing, visual displays. 

Entire general area of communicating 
bibliographic data from files to 
users . 

References to theoretical and/or 
empirical literature on lecturing 
and similar "presentational" methods 
of teaching, the determiners of their 
effectiveness In promoting achieve- 
ment of various educational objectives. 



Person H 



Person I 



Elementary schojol teacher (24, F). 



Secondary school teacher doing 
graduate work to assist him In 
developing media programs at 
his school (26, M) . 



Individualized programs In education, 
flexible scheduling, developmental 
placement . 

8mm single concept films In relation 
to Individualized Instruction. 



Most of the nine had approached the clearinghouse for assistance In 
finding educational Information without knowing of the DIALOG system. A 
few (such as the elementary school teacher and the graduate student) were 
specifically recruited In order to Include persons In those positions, but 






only after prior checking to determine that they had Information needs 
connected with on-golng educational projects. 

Appointments were made about two weeks In advance of the two-hour 
sessions, and this made It possible to send most users the DIALOG manual. 
(Some of the first users could not be supplied manuals as there was a 
delay In printing at Lockheed. Readers of this report who would like a 
copy of the manual should request one from the clearinghouse.) 
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Each user was asked to come about 30 minutes before the time the system 
became operative » and the "Pre-Dialog Questions" (appended) were adminis- 
tered then. The user's questions about the system were answered, and he 
was shown the terminal. When the DIALOG programing was loaded at Lockheed 
and the ERIC document file made accessible on an IBM Data Cell, the visitor 
sat down at the console and performed all searching himself. Mrs. Tlmble 
remained throughout the session, coaching the user and prompting him to 
make use of different aspects of the system until he had some familiarity 
with It. 

After the session was over, and after a brief coffee break, Mrs. 

Tlmble followed the "Dialog Debriefing Outline" (appended) In Interviewing 
the visitor about his experience. The entire Interview was tape recorded 
and, later, converted to a typed transcript. These transcripts were coded 
to produce some relatively objective sunnnary results, and the transcripts 
also provided the verbatim answers presented In this report. 

(Verbatims sometimes are used to "flavor" a report; In this one they 
play a more Important role~they give a good Idea of the evaluators' 
reactions to an Interactive system.) 

Reliability of only 77% was achieved on coding the combined question- 
naires and Interview transcripts, but this was considered satisfactory 
under the circumstances: Mo training program was carried on for coders, 

and most disagreements were the result of two misunderstandings— 

1. In coding the number of good and bad comments about DIALOG, 
one coder Included comments about the general ERIC system* 

2. The other coder counted every good or bad comment, even 
though essentially the same comment had been made earlier 
In the Interview by the same person. 










All nine questionnaires and Interview transcripts were coded by each 
Investigator 9 and then the coding was reconciled by both Investigators 
referring again to the questionnaires and transcripts. 

Some questions on the debriefing outline pertained to the visitor's 
experience with other aspects of the ERIC system. These questions were 
asked last so they could not Influence any other questions, and the answers-- 
though of Interest to ERIC personnel— are not Included In this report 
because they have no direct relation to DIALOG. 

Each of the nine visitors was asked for Information on his previous 
experience with mechanized retrieval systems, so that the relationship 
between that and the evaluation of DIALOG could be Investigated. That 
particular Investigation was abortive; almost all the visitors were 
enthusiastic about the system, and almost all had no previous experience 
with mechanized retrieval systems. (Only two visitors had done more than 
"hear about" on-line retrieval systems. One had read about them, and the 
other was Involved In designing such a system.) 

One conclusion justified by this Information Is that the evaluators 
were representative of the majority of educators who would be served by 
any future move to make the ERIC file available to mechanized search. 
Certainly they were not hand-picked because of considerable experience with 
and Interest In computers and Information retrieval. 
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IV. RESULTS OF THE NINE CASE STUDIES 



A. General User Evaluation of the System 



One big question to be answered was whether Individuals with no 
previous experience with Interactive Information systems could sit down 
at a terminal and. In a reasonably short time, use such a system effec- 
tively. The answer was an unqualified yes. 

For the most part, evaluators were enthusiastic about their two-hour 
experiences. Before they were asked to comment on specific aspects of the 
system, which was a form of prompting, the visitors were encouraged to put 
on record whatever Impressions they wished to report. All of these 
"unprompted" statements are Included In this section of the report. In 
their entireties. In order to give a fair picture of user response. 

The two aspects of the system which were most frequently commented on 
were 1) Its speed, and 2) the way It "widened horizons," the way It 
suggested other relevant areas of Information or different approaches to 
the Information originally sought. 

Following are unprompted statements bearing on speed. Identified by 
which evaluator was speaking: 



" . . .It's just a fantastic savings of time." (A) 

"It would have taken me many months of research to have come 
up with such extensive, comprehensive references." (B) 

"I'm Impressed by the speed . . . ." (C) 

"It very clearly saved me a great deal of time." (D) 
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"... Much easier access to Information .... I would 
never go to the library and search out any of. that ...» It 
creates an Interest, for me." (H) 

"A great saving In time." (1) 

Following are unprompted statements about the "horizon-widening"; 

"... Opened up new avenues for thought.” (B) 

"... Expanding areas that I hadn’t considered as being 
related to the subject." (C) 

"It had a sort of fallout of new Ideas and possibilities." (D) 

"I was amazed at . . . what possibilities It offered for 
further learning.” (H) 

Much of this report attempts some quantification of evaluators' 
i^esponses. At this point, the verbatim transcripts of all general remarks 
are presented because of the feel they give about the sessions. The responses 
to the two first questions (Did you find relevant Information? What are 
your general comments?) are merged here: 

Person A (Researcher engaged In the planning, technical 

aspects and conduct of educational research, 28, M.) 

"Did you locate relevant Information?” 

Very definitely. Of the ones that I've seen, and we saw 
about 30 articles, with the 80-90% that I retained, I would have 
to say that I found relevant articles . . . • The system Itself, 
as you go through It, you're expanding terms and you come across 
terms that Invite a little bit of thinking In other directions 
so It gives you the opportunity to keep expanding your own Ideas 
of what your search Involves, as you actually go through It. So 
I think that's fruitful, too .... I think It's very useful. 
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Just being able to spend approximately two or three hours, to go 
through that thorough a search, has got to be very much to any 

researcher's advantage I think that the system In 

general, the way that It's organized now. Is highly effective. 

I can't think of anything that I'd ^Itlclze. In terms of the 
procedure that Is available to actually see articles, to go from 
seeing the subject-tltles and the authors, etc., to the abstract 
and these types of things, so you can actually ... go Into 
detail or you can leave It alone, all the different combinations 
that you can do with your search— It's Just a fantastic savings 
of time. 



Person B (Motivational educator In private practice, working 
with children referred by schools, doctors, etc., 

Al, F.) 

"Did you locate relevant Information?" 

Very crucial [Information] . And I think, probably. It would 
have taken me many many many months of research to have come up 
with such extensive, comprehensive references .... I think 
what It has done, too. It has opened up new avenues for thought. 
And, you know, creativity. I really do ... . It has great 
potential for releasing some of our rather Inhibited mechanisms, 
you know. In terms of our own ways of thinking. I think that 
somehow, we have been confined to a very very limited way of 
conceiving data. And I think what this Is going to do Is explode 
all of our patterns of thinking, and I think It's going to remove 
a lot of barriers and bring us, probably. Into a new level of 
communication and education and exploration. I Just think It has 
great possibilities for growth .... Just say It has Infinite 
possibilities. 
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Person C (Graduate student in education who will return to 
district to work, 25, M.) 

"Did you locate relevant Information?" 

Yes, I*d say, based on the abstracts that we looked at, and 
on the descriptions of those materials, that I have • • • • 

Yes, the answer Is yes. In one sense, the fact that there Just 
aren’t very many materials In some of those areas""lt*s good 
to find that out too .... I’m Impressed by the speed at 
which you can retrieve Information and particularly by the way In 
^jilch you can combine sets of Information. And then retrieve from 
these sets the relevant factors and the combinations of relevant 
factors. That seems like a very powerful tool to me ... . I 
found a couple of times that In searching broad areas that we ran 
across Information that I wasn’t aware of, and titles and 
subtitles of things I wasn’t aware of. I think this was very 
helpful too, by expanding areas that I hadn’t considered as being 
related to the subject. 



Person D (M.D. engaged In psychiatric research and therapy, 29, M.) 

"Did you locate relevant Information?" 

Not In the exhaustive sense of bibliographical research. But 
It had something else — It gave me some Ideas of new techniques 
that I hadn’t seen before or sort of headings to look under that 
I hadn’t thought of before. So, In the sort of exhaustive sense, 
again, I think It’s probably as useful as giving further references. 
But It had a sort of fallout of new Ideas and possibilities . • • • 
It very clearly saved me a great deal of time. And a great deal 
of energy .... [But] It’s hard on the eyes after a couple of 

hours. 
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Person E (Assistant Professor of Linguistics and Computer 
Science, 27, M.) 



"Bid you locate relevant Information?" 

It's difficult to say, but my guess would be yes. In the 
areas of computers, yes. I realize that the ERIC file Is specific 
to computers as they are used In education, not specifically . 
research In computer science. But, as far as the file goes, yes, 

I would say In computers. As far as linguistics Is concerned, 
partially, I hoped I would get a chance, you know, to look at 
some more of the Information there .... I like the Boolean 
logic In there, as It's very useful. I would say that — first 
let me tell you the things that I like about It, and then I'll 
tell you the things that at least right now appear to be difficult. 
I like the rather direct sequence In which you can go from the 
thesaurus to the actual document, with only three keys. I believe 
It's the EXPAND, SELECT and DISPLAY. The fact that you have to 
do a minimum of typing, that's a very good feature--that you use 
the keys, like EXPAND, rather than type out something. The NOT 
part of the Boolean logic, I wasn't able to see a reason for. 

Now to get Into the critical aspects of It. Having to look from 
the console to the teletype for set number was an Inconvenience, 
but I suppose that one could get used to that. If he were to use 
the system more often .... Aside from that, I was very pleased 
with the system. I think I got what I was looking for 



esse 



Person F (University librarian directing library automation, 
and Involved In developing a different on-line 
retrieval system, 40, M.) 



I'd say. In general, I found It somewhat cumbersome to use. 

I think the fact that you must set aside Index terms, authors, or 
what have you, separately to a set on an auxiliary device, and 
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then do the combining as an after step. Is kind of unhandy. I 
think my preference would be to construct a search the way I want 
It, eventually, to be ... . I think that If the response from 
the cathode ray tube and the whole system In general were a lot 
faster than It was, I perhaps wouldn’t feel constrained by that 
limitation .... I’m Inclined to think a printed thesaurus, 
that the user can have Immediate access to so he could familiarize 
himself with the terms of Interest, would be far more efficient 
for the user. I think after a week or two of use. If a man used 
this dally, even If for as little as an hour a day, he would learn 
his vocabulary, or relearn It, and thus become relatively Independent 
of the printed vocabulary. Having to use the system, to learn 
the vocabulary [the descriptors from the thesaurus?] I think Is 
extremely Inefficient. And very time consuming .... One real 
great feature of this system Is the fact that you do have the sets 
retained on the prlnt-out. If you want to go back to them later, 
you’ve got kind of a backup feature. Your results are saved. I 
think that’s a great advantage, but It’s an advantage that occurs 
after you’ve been at It for awhile — suddenly you realize that you 
want to go back. When you’re first starting out. It looks like a 
disadvantage . 



Person G (Professor of education teaching and doing research 
In educational psychology, 52, M.) 

"Did you locate relevant material?" 

Oh, I’m sure I have, just that one Item that I have copied the 
reference for looked very exciting, and one I have never heard of 
before, but beyond that, whether I really got any gold Is still a 
question In my mind .... I think that the system was working 
along the lines that I wanted It to work. Insofar as the content 
the research and the abstracts that are In ERIC— are relevant. 
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Person H (Elementary school teacher, 24, F.) 

"Did you locate relevant material?" 

Yes, definitely .... Of the two that we researched, I 
thought that they were covered, particularly reading .... I 
think it takes more fami^arization than what I have. I mean, 

I don’t feel I could go in and use it effectively right now, even 
after spending two hours with you, although I think that once you 
. could use it efficiently, then it would be really worthwhile. I 
mean, I was amazed at how much information there was, and what 
possibilities it offered for further learning .... I think, 
because of this efficiency, if it could be easily available to a 
lot of people— once you’ve learned to use it simply, which I 
think possibly is a problem— well, you’d have much easier access 
to information .... I would never go to the library and search 
out any of that, I would never even begin to do any of that, 
because of the simple system like that, you would be more 
encouraged to find out more about your field. I would .... 

It creates an interest, for me. It created an interest in looking 
up things in my field, which I wouldn’t normally do, simply 
because the [regular] process of having to look it up is more 
difficult than I want to spend the time to do. 



Person I (Secondary school teacher doing graduate work to 
assist him in developing media programs at his 
school, 26, M.) 

"Did you locate relevant material?" 

Yes, I did .... I think the system can be very helpful to 
anyone doing research and, particularly, you may narrow this down 
to, you can find out if there is information available in the 
subject you’re researching. This is what I think is a great 
saving in time. One of the greatest helps, I think, is the great 
saving in time. You can devote more time to the research and the 



29 
















study when you have the Information that you need .... It's 
simple to use, I think, particularly after It's explained. After 
you explained It to me, I felt It was very easy to use, and very 
easy to understand. 



B. Good Featvires of the System Singled Out 

* 

After each of the nine evaluators had commented generally on DIALOG, 
he was asked to specify the good aspects of the system (as can be seen In 
the appended ^'Debriefing Outline"). Combining the "general comments" and 
"specific good aspects" statements, there were a total of AA good points 
singled out. (This Is a little like compiling batting averages In Little 
League, but the total score would be AA good points, 28 bad points listed. 
Precisely which aspects of the system were considered good and which bad 
will be reported In this and the next section of the report.) 

Of somewhat greater value than the AA/28 breakdown Is the finding 
that there were 25 different good aspects reported, 18 different bad 
aspects. But there are problems even there; should comprehensiveness, 
effectiveness, and being of general value be considered three different 
favorable aspects? We decided to report them separately, because the reader 
then may want to mentally combine them In a general "effectiveness" category, 
whereas If they were reported In combined fashion the reader never could 
separate them out. The chosen approach has the virtue of leaving errors 
out In the open. 

Six of the evaluators noted the speed of the system and Its saving of 
user time. The next most frequent favorable observations were that being 
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able to combine sets was very desirable (volunteered by 5 evaluators) and 
that using the system had opened new avenues for thought, or "widened 
horizons" (volunteered by 4 evaluators) . Three of the evaluators commended 
the system for being simple to use, easy to work with* 

Five different favorable comments were made by two evaluators each: 



The system Is thorough, comprehensive. 

The system Is effective. 

The system Is generally valuable. 

The system Is exhilarating. Interest-creating. • 
Having the hardcopy record to refer to Is handy. 



The favorable aspects of the system volunteered by single evaluators, 
along with those already reported, are Included In the table below* The 
letters Indicate which evaluator or evaluators made each comment: 
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User 

ABCDHI 

CDF6I 

BCDH 

CHI 

AH 

AH 

AH 

BH 

EF 

B 

B 

B 

B 

C 

C 

C 

D 

E 

E 

E 

E 

E 

E 

E 

F 



Favorable Aspects of the System 
Volunteered by Evaluators 

Statement 

The system has great speed, saves user's time. 

Being able to combine sets that way Is a desirable feature. 

The system opens new avenues for thought. It "widened horizons." 

It Is simple to use, easy to work with. 

It Is thorough, comprehensive. 

It Is effective. 

It Is, In general, valuable. 

It exhilarates, creates Interest. 

Having the hardcopy record to refer to Is handy. 

The system releases Inhibiting mechanisms. 

It brings us Into a new level of communication and education and exploration. 
It Is challenging. 

It provokes you to order your approach. 

It presents good descriptions of documents. 

It presents a variety of topics. 

There Is the attractive possibility of being able to access the system 
from anywhere In the country. 

The system saves user energy. 

The Boolean logic Is useful. 

The direct sequence of going from the thesaurus to the document Is nice. 

There Is a minimum amount of typing, because of the presence of the 
command keys. 

The particular documents sought were Identified. 

[It Is a generally attractive system] — ^^"we'd like to have one." 

Presented as they are on the CRT, the terms can be scanned quickly. 

Paging was easy. 

The expand feature was useful. 
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C. Bad Features of the System Singled Out 



As can be seen In the apg^anded "Debriefing Outline," the evaluators 
were prompted to Identify what they considered bad features, but they 
were not asked leading questions. The conments presented here are from 
the combined "general comments" and "specific bad comments" statements. 

Delays In waiting for the system to accept and execute a command were 
singled out as a bad feature by four evaluators, as was the feeling that 
considerable experience or time was needed to master all the operating rules. 
Three different critical comments were made by two evaluators each: 



Too many combinations of keys are needed to Input one command. 



Having to build combined sets one step at. a time, rather than 
using parentheses, and doing It with one complex statement Is 
Inconvenient, and 



There Is a great deal of "paging" required on the CRT, because 
you can only look at nine terms at a time. 



The unfavorable aspects of the system volunteered by single evaluators, 
along with those already reported, are Included In the table below. The 
letters Indicate which evaluator or evaluators made each comment: 
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Unfavorable Aspects of the System 
Volunteered by Evaluators 



User 



Statement 



ABFI 

FAIC 

AI 

EF 



EF 



B 

D 



D 

E 



E 

E 

F 

F 



F 

F 

G 

I 



There are delays when you must wait for the system. 

Considerable experience or time Is needed to master the rules. 

00 many combinations of keys are needed to Input one command. 

Having to build combined sets one at a time, rather than doing It with 
one complex statement. Is Inconvenient. 

There Is a great deal of "paging" required on the CRT, because you can 
only look at nine terms at a time. 

The Innovative system can be frustrating for people who don’t like 
to change. 



The system Is limited because of the original Input [a reflection not on 
DIALOG, but on ERIC] . 



The CRT Is hard on the eyes after a couple of hours. 



Having to look back and forth from the CRT to typewriter unit Is 
Inconvenient . 



Having to repeat pushing the button so much~for every 10 things you want 
printed— *ls Inconvenient . 

A hierarchical thesaurus display would be preferable. 

It must be expensive. 

It Is cumbersome to use. 



Having to use DIALOG to approach the thesaurus Is Inefficient [actually, 
of course, you don't]. 



The letters on the CRT have an unpleasant shape 
The CRT display Is "painted on" too slowly. 

There are weaknesses In Indexing [this Is not a weakness In DIALOG]. 



The hardware falls [some set numbers were not printed out during that 
particular session] . 



The touch of the keyboard Is too different from that of most typewriters. 
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D. Prompted Evaluation of Components of the System 

Each evaluator had unstructured opportunities to specify what he 
considered bad or good about the system, and these comments formed the basis 
for parts A, B and C of this section of the report. Then he was specifi- 
cally asked for comments on the different components. One problem with 
this approach was that some comments were on the hardware, while others 
were on the process of Its use. Both kinds are presented here. 

1. The Terminal Keyboard 

Most evaluators found no fault with the keyboard, but four did wish 
that search and display commands could be entered with less complex key 
selection. 

Unfavorable comments (with letters Identifying which evaluator 

or evaluators made each comment) : 

ACGI Being able to use one key, rather than shifting for 
commands, would be desirable. 

FI Keyboard doesn't have good touch,, seems clumsy. 

Favorable comments : 

ABCDG Keyboard Is easy to use. Is a universal type thing [so Is 
easy to get used to] . 

E Keys are closer than on big IBM 2741--that's one good 

feature, that you don't have to move around too much. 

H Isn't a universal size — Is smaller — but doesn't present 

any problem. 



35 








I 



I 

i 






2. Typed Terminal Record 

Only two relatively minor criticisms of the typed terminal record were 
forthcoming. Four evaluators described it as generally useful, four said 
that it was valuable after the search, as a record. 

Unfavorable comments t 

. A Having to keep turning the knob to get the paper up to 

separate some of the concepts is undesirable. 

D I forgot what some of the letters stood for. 

Favorable comments ! 

BFGI Generally useful. 

CEHI Is valuable later on, as a record. 

Cl Is useful as a self-correcting device, during the search. 

DH Is easy to understand. 

E Can be used later to explain how a system like this works. 
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3. The Cathode Ray Tube Display 

Most evaluators felt the visual display on the cathode ray tube allowed 
great savings of time, and was clear and easy to read. Other favorable 
comments were that it was exciting and that that mode of presentation 
avoided the clatter of a printer. On the other side of the ledger, some 



evaluators found awkward features: 



Unfavorable comments: 



D 

G 



Gave me bleary eyes after a while. 



Breaking words at the end of the line, with no regard 
for syllabification, is awkward. 
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Sometimes It's hard to line up numbers, to figure out which 
line a number Is on, because they're close together. 

H In going through displays, after you hit the KEEP command, 

and the ENTER command, why couldn't It just naturally 
progress to display the next one? 

I It takes a few minutes to get used to. 



Favorable Comments ; 

ACEG It saves lots of time. 

CHID It was easy to read, clear. 

BH Very clear, very exciting ... a very natural way of 

receiving Information. 

D Very helpful . 

D Being able to see the abstract that way was extremely 

helpful . 

E Avoids clatter of a printer. 

F For a quantity of data, you must have a visual display . . 

. . [This was] a nice combination of the two kinds of display. 



4. Timing of System Response 

Evaluators were specifically asked about this because some clearinghouse 
personnel came to be critical of delays when It was necessary to wait to 
Input the next command. Only four of the nine evaluators were at all 
critical of delay; most were so Impressed by the performance of the system 
that any de.lay was of no consequence. One felt such variation In pacing 
suggested that he was running the machine, rather than vice-versa. Though 
the logic of that last remark Is tenuous, the general conclusion was 
evident: At least while learning to use the system, few persons are bothered 

by having to wait sometimes to enter commands. 
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Unfavorable tommehte ; 

F Delays bothered me In paging. 

H It took time to get accustomed to waiting to enter the 

command to move to the next display. 



Favorable Comments ; 

AGI . . . You realize It's Inevitable, you've got to have 

some sort of delay, because you have a tremendous system 
going to work here .... 

B No [bother] . I was so busy, thinking about what was 

happening . 

C No [bother] ... It's just as well that there may be 

some variation in pacing. [Otherwise] I think I'd get 
a little nervous about am I running the machine or Is 
the machine running me? So I don't mind the pacing. 

E Delays bothered somewhat . . . but the pause Is shorter 

than the WYLBER (another system) pauses are. 

5. Having the Thesaurus Available on the System 

Prompting the evaluators to comment on the thesaurus produced little 
criticism. All thought of It as a useful feature. 



Unfavorable comments ; 

F Would possibly be less expensive [to have It available only 

In hardcopy] , or have only the updated terms ... on the 
terminal . 

Favorable comments ; 

ABDEFGHI Fine, useful. Important, no criticism. 

CD Helps you to zero In on things you may have missed. 

It seems fair to point out here that the existing ERIC thesaurus was 
not being compared with other thesauri, or with any might-have-been ERIC 
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thesaurus. The evaluators were responding to having the thesaurus available 
for machine searching, rather than depending on the hardcopy version. 

6. The Manual 

Like the thesaurus, the manual came In for few negative comments from 
the evaluators. Clearinghouse personnel felt It should have been more 
oriented to the educational user (as Is pointed out In section VI of this 
report) , but only one of the evaluators criticized It . In addition to 
calling part of the manual confusing, that evaluator made what was considered 
an excellent suggestion: Computers should not be used as the subject of a 

sample search. In the manual, when the retrieval system which Is being 
Introduced Is Itself computerized. 
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Unfavorable comments: 



It started out to be easily understood, got very confusing. 
I think the last part could be rewritten In clearer terms. 
And ... I think It definitely should not be dealing 
with the subject of computers .... This Is confusing 
because you're using a computer. 



Favorable comments: 



I didn't find the manual difficult. 
Instructions quite good. 



I thought the 



B 

DH 



It's clear, complete. 



O.K. 



The other four of the evaluators did not comment on the manual, either 



because printing delay at Lockheed kept the evaluators from being supplied 
prior to arriving for their appointments or because the question was 
Inadvertantly omitted In their debriefing Interviews. 
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E. Reconsideration of the Unfavorable Comments 
in View of Later Changes 

With the implementation of a new version of the programing, called 
DIALOG II, on July 29, 1969, certain of the features commented on unfavor'^ 
ably were improved. The following contains selected unfavorable comments 
and a description of the new features which bear on them. DIALOG II was 
introduced after the last evaluation session had been conducted, but some 
data on DIALOG II is discussed in section V of this report. 

"Having to build combined sets one step at a time, rather than doing 
it with one complex statement is inconvenient." Previously, only one type 
of Boolean operation could be done with one COMBINE command. It will be 
remembered that there are three operations, OR, AND, NOT. With DIALOG II, 
through the use of parentheses, the set numbers could be grouped in complex 
expressions so that more than one operation can be performed at once if 
desired, i.e., ^ (l+2+3+4)*( 7+12+13+14) . 

Another new feature made it no longer necessary to SELECT items from 
an index or thesaurus display one at a time. Ranges of items could be 
selected with the same command, using the comma and hyphen as punctuation 
for grouping the terms. For example: <E5-E7,E10,£12 selects out items 
E5 E6 E7 ElO and E12. The one limiting factor here is that you must want 
these sets "ORed" in your final product, as only one set is created by the 
selection of such a range. 

"There is a great deal of 'paging' required on the CRT, because you 
can only look at nine terms at a time." While this criticism was not 
entirely obviated, the displays of the indexes and the thesaurus entries 
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were differently arranged to allow more than nine terms to appear at one 
time. The number was no longer constant; the number that could be seen at 
one time depended on the length of the terms being displayed. The maximum 
was 18 terms. Paging was no longer accomplished by pressing the ENTER key; 

It was a separate command. Finally, It was possible to PAGE forward In 
the Index display. Instead of having to EXPAND a new term In order to see 
what follows the ninth term on the display. (It was always possible to 
PAGE forward In the thesaurus display.) 

"Having to repeat pushing the button so much — for every 10 things you 
want printed— Is Inconvenient." This feature was originally Intended as a 
safety feature to keep someone from accidently triggering a print of a 
huge set of documents. Since sets of 1,000 or 2,000 documents are frequently 
created and the entrance of a wrong set number with a PRINT command could 
be disastrous, this was a good Idea. However, we soon realized that we 
would be printing larger sets of documents and doing more frequent prints 
than previous Implementers of the system For these reasons, the number 
of Items which could be printed with one command was Increased to 50 In the 
new program. 

Additional features made available Included a specific message command 
and a command which would display the set history, or record of sets and 
what they contained, on the CRT. 

F. Serendipitous Effects of Using the System 

As already reported, the hands-on sessions were successful in locating 
relevant information for the nine, visitors. The question or questions which 
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But two other features of the 



they brought to the session were answered, 
system were evident: Users were led to ask additional questions about the 
chosen area of Investigation, and to pursue entirely different matters than 
those which originally concerned them. 

This section of the report deals with the "Intellectual fallout" or 
"horizon-widening" effect of the system. The section after this one deals 
with the advantages of the user Interacting with the machine In planning 
search strategy, which — although a related aspect — Is treated separately. 

Seven of the nine evaluators reported that they asked additional and 
different questions about the subject which they originally were Investigat- 
ing, and seven — although not the same seven — said that they came upon 
material on different, though related, subjects which they would like to 
pursue at a later date. 

Verbatims are presented here to demonstrate user reaction to this 
aspect of the system: 



A I began to realize they had some articles of International 

scope . . . [and that] kind of opened up that area .... 

C Well, as we began to look at the section on Instructional 

television, there were some related topics there that I 
hadn't been aware of ... . There were a couple of topics 
that Interested me, one In the area of teacher training, 
which I Just happened to run across, but I would like to 
look Into at another time, although It wasn't particularly 
related to this study. 

D The only thing I formulated was new areas to look under, for 

relevant research. 

E Let me tell you some questions that were answered that I 

posed to myself, and were answered during the process . . . . 
New questions that are already answered .... I stumbled 
on something, again because there are some things that are 



not In the thesaurus but are In DIALOG [identifiers]. I 
stumbled onto [specific] programing languages. 

H ... The biggest problem was staying with what I had 

originally pursued, Instead of getting off on other 
interesting things. 

I I did run across some other information [that I would 

want to pursue later] .... We've got our mind on one 
thing, particularly when we're researching something, 
one phase, and we only think to look in certain areas, 
and the thing that I liked about this, the machine points 
out that there may be some other related information . . 

. . This is, I think, very significant, and very helpful. 



We assume that such "horizon-widening" effects are extremely desirable, 
and we can imagine no research administrator with such a narrow area of 
concern as to object to this aspect of DIALOG. To look at the situation 
from a different viewpoint, this aspect tends to bring the useful documents 
in the collection to the user's attention even though his preconceived ideas 
of what is available are Incorrect. 

This leads to the next section of the report, which evaluates being 
able to keep tabs on the search, and modify It, as it progresses. 



G. Advantages of Interacting With the Ongoing Search 

Six of the evaluators had favorable comments on the way the system 
made It possible to monitor and modify searches, but the other three should 
not be considered negative or indifferent. Most of the evaluators had no 
previous experience with computerized retrieval systems, and so could 
hardly compare DIALOG with non-lnteracclve, batch-processing systems. Our 
feeling was that the evaluators reacted to DIALOG as an entity, and that 
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the overwhelmingly favorable general comments were to a great extent the 
result of this very basic aspect of the system. 

The Individual verbatims: 



A It makes a big difference, because you get a feeling of 

control over your search that you don't have so much 
when you're actually In the library. There It's hard to 
remember exactly which things you were going to go back 
and do — you have to write things down, you have to 
organize things. Here you have handy little systems for 
putting something off somewhere and you just organize In 
your own mind the very basic concepts. 

B It has a much more organizing effect. It helps to organize 

In a much more effective way. 

C It helped me ... . I think the fundamental help was I 

had some Idea of the amount of Information I was handling, 
or would be handling If I had It printed out. And It 
gave me some Insight, too. Into the amount of research 
that had gone on In certain areas. 

D It's like having a great mass of Information at your 

disposal, where you can somehow set up and know where you 
are and how much you've looked at. 

F I think It made a great difference. 

I It gives you other Ideas and other, related, sources. 







V. RESULTS OF THE 19 LATER EVALUATIONS 



After the nine intensive case studies had been completed, the 
retrieval system was used to conduct searches requested by visitors to the 
clearinghouse. The observations and reactions of 19 of those visitors 
were recorded on six-page questionnaires which paralled, to a great extent, 
the "debriefing outline" used in the nine case studies. By the time the 
first of the 19 visitors arrived on the scene, the previously discussed 
modification of programing had been completed and it was DIALOG II rather 
than the original system which was observed. 

There were no striking differences between the responses recorded on 
questionnaires by the 19 and the tape-recorded comments of the nine 
evaluators. One question on the questionnaire, however, had not been put 
to the nine evaluators: 



All 18 visitors who answered the question indicated that they would like 
to use such a system several times a year or oftener. A third of the 
visitors indicated they would like to use it once a week or oftener. The 
number giving each answer is shown below: 



If DIALOG were readily available to you at a convenient 
location, how often would you like to use it? 



several 



more often 



never 



once a times once a 

year a year month 



once a than once 
week a week 



0 



0 



6 



6 



4 



2 
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About half the visitors (8 of 18) indicated the system revealed new 
aspects or dimensions of the subject to them, and about half (10 of 19) 
reported coming upon material on an entirely different subject which they 
then wished to pursue. 

The 19 visitors were asked to volunteer what they considered good and 
bad features of the system. A total of 65 good features and 16 bad features 

i 

were listed. No novel comments not Included in the earlier remarks of the 
nine evaluators were made. 

When prompted to evaluate specific features of the DIALOG system, the 
ratings were uniformly favorable. They are presented here In the spirit of 
making the record complete rather than with the Idea of surprising any 
reader. Each scale Is presented as a continuum, from the descriptive term 
presented at the left to that at the right. The number of visitors checking 
each point on the scale is indicated: 



The Keyboard 



easy 
to use 



difficult 

6 6 4 2 0 to use 



The Typed Terminal Record 



easy 
to use 


6 


9 


3 


0 


0 


difficult 
to use 


valuable In 
conducting 


17 


0 


0 


1 


0 


not 

valuable 


a search 
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The CRT or Visual Display 



easy 
to use 



10 



difficult 
to use 



valuable In 
conducting 18 
a search 



not 

valuable 



The Thesaurus 



easy 
to use 



8 



difficult 
0 to use 



valuable In 

conducting 7_ 

a search 



not 

0 valuable 
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VI. REFLECTIONS ON HAVING THE SYSTEM 
AVAILABLE FOR SEVERAL MONTHS 



t 



Besides the nine case studies presented In this report, there are some 
observations on DIALOG which are a product of Its use In the clearinghouse 
for a variety of tasks. These applications can be categorized as duplicate 
checking, ln~house projects, answers to Information requests, and visitor 
use. The actual experience which was gained from using DIALOG for these 
purposes has bearing on any future computerization of the ERIC files. 

Duplicate checking Is an extremely time-consuming process which. In 
the past, has Involved manually checking each acqulsltlcned document against 
the current acquisition data log, and all past Indexes. DIALOG has made 
It possible for the staff member In charge of this processing to do the 
checking more quickly and with greater assurance of not missing a document. 
The only requirement for this Is that the file be as current as possible. 

A real Improvement would be the ability to access the acquisition data log 
on the CRT, thereby streamlining the process still further. 

A spin-off of the system Is a "dump" of the Indexes. This listing 
makes It possible to do routine checking at any time, saving on-line 
processing for the especially difficult documents. Again, for this to be 
continuously effective. It would be necessary to keep the listing updated. 
This listing also has value as a record of the number of relevant entries 



In the file on any given attribute, which makes It possible to check the 
size and scope of any search before actually getting on the system. 

During the last two months of the contract period an updated file of 
20,000 documents was being searched (the ERIC files through August of 1969) 
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Due to problems of disc space a title Index was not available, which 
rendered the system nearly useless for duplicate checking. Some additional 
special programing or more disc space would solve this difficulty. 

The in-house aspect of DIALOG use is similar to visitor use in that the 



system can be used to obtain extensive bibliographies on any subject being 
considered for a locally produced bibliography or state of the art paper, as 



well as to do literature searches to support other projects being undertaken 
by a clearinghouse. Three clearinghouses (in addition to ERIC at Stanford) 



have made use of this facility to produce long bibliographies for various 



projects. It has also been used to produce supporting bibliographies for 
clearinghouse-affiliated people working on a project of the Institute for 
Educational Development, evaluating the impact of Title VII research. Thus 
clearinghouse activities related to the analysis of information in the 
clearinghouse's specialty area are greatly assisted by the on-line searching 



capability. 

The load of quite specific informational requests which flow into the 
clearinghouse (approximately 40 per month) can also be handled more 



effectively by using DIALOG. While it was in operation, the clearinghouse 
staff used DIALOG to handle 93 of these requests in depth. Without the 
on-line facility it would have been Impossible to give each of these users 



an exhaustive bibliography. The requests instead would have been handled 



in a general way, perhaps by giving the person information on where he 
might start his search or who might be able to help him. DIALOG made it 



possible to give the requestor listings of things already processed by ERIC 
which conformed to his particular interests. Feedback from people receiving 
this service indicates that it was extremely useful and greatly appreciated. 
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An off'Shoot of handling these requests was a compilation of the duplicates 

I' 

I of past searches; these can be used to answer similar requests without 

i using additional computer time. 

Finally, and most Importantly, DIALOG proved valuable to clearinghouse 
visitors. Through October 31, 46 people had had a demonstration and Intro- 
duction to the system, 68 people had had their requests searched by a staff 
I 

I * 

I member while they were present to Interact with the system, modifying the 

I 

I search strategy as necessary, and 21 people (Including the nine evaluators) 

I were taught to use the system and had hands-on experience. These people 

I 

I who used the clearinghouse as a source of ERIC materials were able to do 

I 

I their literature searches efficiently. 

I 

I Perhaps most Important here Is the ease of use of the system. For . 

p 

I 



people who are unfamiliar with computers and who have only a limited amount 

t 

I 

I of time to devote to their professional research and to learning to use a 

a 

I 

I new research technique » no matter how powerful It may be» It Is quite 

i' 

I Important that the technique be simple to understand. Experience In 

I demonstrating DIALOG and Instructing people In Its use Indicates that It 
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Is simple and does not overwhelm the person unfamiliar with computers. The 
Instruction did require staff time to teach people on an Individual basis, 
but It Is clear that the process could be made basically self -Instructional 
through the development of some materials and a film or slide presentation. 
Then a minimal amount of staff time would be required to monitor the first 
use of the system, until the user was proficient. 

To assist In explaining DIALOG, a set of 27 slides was prepared to 
demonstrate a fairly Involved search for ethnic studies materials. A copy 
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of the script for the slide presentation Is available free from the 
clearinghouse, and a set of the slides can be purchased from the Information 
Projects Research Fund (Attn. C. Collins, ERIC at Stanford) for $10. 

Under the system employed. It took 20 to 30 minutes to familiarize 
the person with the equipment and procedures, after he had read the maaual, 
and an additional half hour to an hour for him to become fairly adept at 
using the system. (Two of the evaluators reported a lack of confidence In 
their ability to use the system, even after two hours, but their results 
Indicated they were effectively conducting searches.) Thus, with a small 
Investment of time (no more than Is ordinarily spent Introducing visitors 
to the general ERIC system) a very powerful means of searching the file Is 
made available. 

Related to the question of Instructing people In the use of DIALOG Is 
the question of the manual. While It was useful. In general It probably 
was not adequately adapted for the educational user. Substantial revision 
could make It a more Informative and Instructive tool for the person 
unacquainted with computer systems. 

It Is Interesting to note that the real difficulty In teaching people 
to use DIALOG had nothing to do with the system Itself. Rather, It was 
the concept of coordinate searching that proved to be difficult. If the 
Individual understood how coordinate Indexing worked. It took only minutes 
to acquaint him with the few mechanical procedures which would allow him 
to search the file that way. However, the linear method of searching out 
materials Is Ingrained In most people, and time Is required to help them 
understand coordinate searching. 
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No real pattern In search strategies emerged In the nine case studies. 
There are several reasons for this, one being that establishing a pattern 
requires extended use of the system. The other Is that the nature of the 
search question rather than a personal pattern seems to determine the 
search strategy. In fact, as the Interviews Indicated, the display of 
thesaurus terms and of sets of documents retrieved by a strategy often 
opened up new approaches or dimensions to the Initial question. 

Also of Interest Is the fact that a person's satisfaction or dis- 
satisfaction with his DIALOG experience related not so much to the procedures, 
but to whether the ERIC collection contained good sources of relevant 
Information. Thus the concern was not with a difficult technique but with 
the quantity and quality of the literature being made available. 

One matter of concern In the experience with DIALOG was Its reliability. 
While this should not be overemphasized, there was a certain undependablllty. 
This came from a variety of sources and unfortunately was additive. There 
was some difficulty with hardware which, of course, was the responsibility 
of the supplier (In this case IBM) . There was difficulty from time to time 
with the programing Itself, which also caused delays and down-time. In the 
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context of only six hours per week of on-line time, these delays and diffi- 
culties caused problems and frustrations for those using the system. 



During the 'time that a second DIALOG terminal was operational at the 
U.S. Office of Education Regional Office In San Francisco, there were 



additional delays at Stanford caused by phone line problems with the San 
Francisco terminal. The equipment (CCI) used at San Francisco had the 
advantage of being less expensive that the equipment at the Stanford 
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clearinghouse. This decrease in expense was related to the type of phone 
connection; for the CCI equipment a standard audiograde phone line was used. 
Some of the delays were directly attributable to this lower quality 
connection between terminal and central processor. 

As an estimate, once every two weeks (six sessions, or twelve hours) 
there was serious difficulty, at the Stanford clearinghouse, with the system 
It is reasonable to assume, however, that increasing the size of the service 
blocks would not result in a proportionate increase in down-time. In a 
context which did not require tight scheduling and prior appointments, and 
where there was ample computer time available to do the things which needed 
to be done, this would seem less Important. 

An additional relevant fact is that over time the "bugs" in a program 
tend to be smoothed out, and so the system becomes more reliable with use. 
One reason we did not experience this kind of increasing dependability is 
that a considerable number of new and useful features were being added, 
and the actual program itself was changed once (see section IV-E of this 
report). The second program, DIALOG II, has additional sophistication and 
some procedural options which make it more powerful. 

Only one further comment is in order: In any situation such as ERIC's 

I 

project with DIALOG, cooperation between the people and groups Involved is ‘ 
most Important. The Information Sciences Laboratory Staff at Lockheed, and 
particularly Roger Summit and Don Schaaf, were extremely helpful. Their 
assistance and good will— and that of James Prevel, Acting Chief of the 
Bureau of Research's Equipment Development Branch — were most appreciated. 
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Name 



questionnaire ~ Part I (Pre-DIALOG Questions) 

1. Please describe briefly the Information you will be looking for in the ERIC file: 



2. About how many documents do you expect to find in the ERIC file on this topics 
(Please circle the correct answer) 

10-25 26-50 51-100 More than 100 



Less than 10 



3. Have you ever used the 

yes no (If No> go to Question 5) 

Cjf YES:) Have you searched RIE by subject? 

Have you searched RIE by title? 
Have you searched RIE by author? 



ERIC publication. RESEARCH IN EDUCATION? (Circle yoUr answer) 



YES 


NO 


YES 


NO 


YES 


NO 



5. Have you chosen any terms 

(If YES;) Please list the principal ones: 



from the ERIC Thesaurus to begin your search? 

YES NO 



HOW familiar are you with on-line retrieval systems? (PLEASE CHECK EACH ANSWER 

THAT APPLIES) 

( ) I*ve heard about them 

( ) I*ve read about them 

( \ T Qtiiidied them in some detail 

( ) I've designed or cooperated in the design of an on-line system 
( ) None of the above 

( ) I have watched another person use _ 

( ) I have had no previous experience with DIALOG 



8. Have you read the DIALOG manual? 



YES 



NO 
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How much experience have you had with on-line retrieval systems other than DIALOG? 

(PLEASE CHECK THE ANSWER THAT BEST APPLIES) 

( ) I have used several systems » each on more than one occasion 

( ) I have used several systems » each on one occasion 

( ) I have used one system on more than one occasion 

( ) I have used one system on one occasion 

( ) I have watched others using system (s) 

( ) I have had no previous experience with on-line systems 









DIALOG DEBRIEFING OUTLINE 






(Questions calling for a prompt 
of additional Information should 
be reprompted until the responder 
can't think of anything else.) 



Before using the system, you said you would be searching the file for Information on 

, I realize that you haven't had 

a chance to check the full texts of anything you retrieved, but do you think 
that you have located some relevant Information? 

Were your original questions answered to your satisfaction? 

What are your general feelings or comments on the system? 

(anything else?) 

What would you pick out as Its particularly good features? 

(anything else?) 

What are Its bad or awkward features? 

(something else?) 



I have already asked you what questions you came In with. What, If any, new questions 
did you formulate during the search? 

(any others?) 

What led you to ask those questions? 

How well did those questions get answered? 



Did you run across any other material — perhaps entirely different subjects — ^whlch 
Interest you and which you might want to search later? 



Using the printed terminal record to help you remember specific things, I'd now like 
you to go back over the search explaining the strategy we used, the terms we 
employed, how they were combined, what results you were getting, and what strategy 
changes you made and why. Sort of a play-by-play. 



Have you any other good comments on the system? 
(something else?) 

Have you further bad comments? 

(anything else?) 
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progress of the search? 

(any other differences?) 

HOW did this affect the search? (if not already clear) 



How do you feel about: 

the keyboard? Was it awkward, easy to get accustomed to? 
the typed terminal record? 

the CRT, visual display? What about this as compared to a single line or lines 

on a typewriter terminal as a means of monitoring 

the search? 

the delays? Did they bother you? 
the thesaurus? 

the manual? Was it clear, useful? 

Not limiting yourself to DIALOG, but to the whole ERIC system: 

What good things could you see? (anything else?) 

What bad things or things that need improvement? (anything else?) 



u 



I 




What about the indexing? 

Was it adequate? 

Are the terms appropriate to the information? 



Do you think that the range of information on your subject is covered as complelely 
as it should be, or is it only adequately covered or insufficiently covered. 

Have you any suggestions as to the way this could be improved? 



o 

ERIC 



1 
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What about microfiche as a format for the storage of information? 

Have you ordered things — fiche or hardcopy — from the ERIC Document Reproduction Service 
in the past? 

How was the service? 

How useful would it be to you to get hardcopy of selected documents in two weeks at 5<: 
a page? (assuming that there is no problem with finances) 

How useful Is hardcopy delivery in half an hour if it Is lOo a page? 

Which would you prefer? (If not already obvious) 









SEARCH TITLE: 
DATE: 

REQUESTOR: 



ADULT BASIC EDUCATION PROGRAMS 

07/11/69 

PERSON A 

SET NO. IN DESCRIPTION OF SET 



COMMAND-OPERAND(S) NO. 
E-IT/BASIC ADULT ED 



SET (+-OR , *-AND , —NOT) 



The 4 Qolwm headings indicate what 
command was given^ what set number 
was assigned (if a set was created) y 
how many documents are contained 
in that sety and finally a descrip- 
tion of its contents. 



The index around the descriptory BASIC ADULT 
EDUCATION y was expanded (E), No entries were 
posted to this term. 



S-E9 
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IT/BASIC READING However y BASIC READING was there and 

was selected (S) to form set 1, 



E-E9 



BASIC READING was then expanded by its reference 
number y E9y in order to display its thesaurus ' 
entries. There was nothing of interest. 



E-IT /ADULT BASIC ED 








ADULT BASIC EDUCATION was 


S-E5 


2 


126 


IT/ADULT BASIC EDUCATION 


expanded and then selected 
to form set 2, 


E-E5 






It was expanded again by the reference number to 








show the thesaurus entries. 


S-E12 


3 


57 


IT/LITERACY EDUCATION The 


term LITERACY EDUCATION was 








found there and selected. 


E-IT/REMEDIAL PROGR 








Nexty REMEDIAL PROGRAMS was 


S-E3 


4 


63 


IT/REMEDIAL INSTRUCTION 


expandedy and it and a number 


S-E4 


5 


4 


IT/REMEDIAL MATHEMATICS 


of alphabetically related 


S-E5 


6 


61 


IT/REMEDIAL PROGRAMS 


terms were selected (sets 


S-E6 


7 


58 


IT/REMEDIAL READING 


4 through 9), 


S-E7 


8 


6 


IT/REMEDIAL READING CLINICS 




S-E8 


9 


22 


IT/REMEDIAL READING PROGRAMS 


E-E5 






REMEDIAL PROGRAMS was then expanded by 








reference number. 




S-E14 


10 


63 


IT/COMPENSATORY EDUCATION 


Two more terms (sets 10 and 


S-E17 


11 


94 


IT /EDUCATIONALLY DISADVANTAGED 11) were located vn vts 










thesaurus entries , 


E-E14 








COMPENSATORY EDUCATION 


S-E18 


12 


145 


IT/COMPENSATORY EDUCATION PROGRAMS was then expanded its \ 






I 






s( 



I 



and one more relevant ter 
was located in its 
thesaurus expansion 
(set 12), 
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I 



I 
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i 



I 



E-IT /ADULT FtJCATIO 


S-E2 


13 


8 


S-E5 


14 


231 


S-E8 


15 


117 


E-IT /ADULTS 


S-E4 


16 


184 


S-E6 


17 


1 


S-El 


18 


40 


S-E5 


19 


87 


E-IT /BASIC SKILLS 


S-E5 


20 


50 


C-3-12/+ 


21 


504 



The searcher now turned to 

IT/ADULT DEVELOPMENT the adult aspect or concept of 

IT/ADULT EDUCATION h,is search. Expanding ADULT 

IT /ADULT EDUCATION PROGRAMS EDUCATION produced three relevant 

terms (sets IS, 14, 15), 



IT /ADULT VOCATIONAL EDUCATION 
IT/ADUT VOCATIONAL EDUCATION 
IT /ADULT STUDENTS 
IT/ADULTS 



Expanding ADULTS 
produced four terms to 
be selected (sets 16 
through 19), 



IT/BASIC SKILLS 



And finally^ BASIC SKILLS^ which relates 
to the earlier concept^ was expanded 
and selected. 



I 



i 



i 



1 



3+4+5+6+7+8+9+10+11+12 Set 21 was created by the union or 

addition of sets 3 through 12, 

This set then included most of the 
remedial or basic education terms. 



C-1+20+21 



22 



595 



1+20+3+4+5+6+7+8+9+10+11-?-12 Set 21 was then added to sets 

1 and 20 to create the basic 
education concept group contain- 
ing everything in the ERIC files 
indexed by one of these terms. 



C-13-15/+ 

C-23+18+19 



23 



352 13+14+15 



Set 23 was created by adding some of 
the adult terms together. 



24 456 18+19+13+14+15 



This process was completed by the^ 
addition performed in set 24^ which 
now represents the adult concept in 
the search. 



Note that so fan set 2 has been ignored^ because it pre-coordinates 
the two concepts in the search and therefore should not be included 
in either coneept set. 



C-22*24 



25 



39 



( 1+20+3+4+5+6+7+8+9+10+11+12) *( 18+19+13+14+15) 

Sets 22 (the basic or remedial education concept) 
and 24 (the adult education concept) were next 
combined (C) to form an intersection^ the AND 
in Boolean logic^ with the resulting set 26 
containing 39 items indexed by at least one term 
from each concept set. 



iifi 

I 



o 

ERIC 
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D-25 

K-25/1 

K-25/2 

K-25/3 

K-25/4 



S-IT/ ILLITERATE ADU 26 



C-26*22 27 



C- 25+27 



28 



D-28 

K-28/2 

K-28/7 

K-28/8 

K-28/10 

K-28/12 

K-28/13 

K-28/14 

K-28/16 

K-28/18 

K-28/19 

K-28/20 

K-28/21 

K-28/23 

K-28/25 

K-28/26 

K-28/27 

K-28/28 

K-28/29 



set was t?zen displayed (D) and the f^rst 
four retrieved items examined one by one on the 
CRT, The relevant ones^ in this case all that 
were examined^ were set aside using the keep (K) 
command into a reference set for future attenUon, 
This reference set is arbitrarily numbered 99, 












33 IT/ILLITERATE ADULTS 



In the examination of the first 
four items of set 26 ^ a new term 
was turned up, ILLITERATE ADULTS, 
which had not been located earlier. 
This term was now selected directly 
(without going through the expansion) . 



21 



(1+20+3+4+5+6+7+8+9+10+11+12) *26 

The resulting set 26 was comb%ned by an AW 
operation with set 22 (basic education terms) 
to form set 27, 



57 



((l+20+3+4+5+6+7+8+9+10+ll+12)*(18+19+13+14+15))+((l+20 

+3+4+5+6+7+8+9+10+11+12) *26) 

The results of this combinaUon and the prev^ous 
one (set 26) were then added together to^ form^ 
set 28, Note that the number of items in 28 ^s 
not equal to the sum of the items in sets 26 and 
27, This is so because a combination creates a 
set of unique documents where no item is 
repeated a second time. 



t 



i 






I 



I 



i 



Set 28 was then displayed 

item by item and the relevant ones 

set aside in the reference set. 



1 



■I 
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K-28/30-57 



C-2-28 



After examini-ng SO references and 
keeping 22 of them^ the evaluator 
determined to keep all of the remaining 
17 and proceed further with the search, 

29 109 2 - (((1+2CH-3+4+5+6+7+8+9+10+11+12)* (18+19+13+14+15) )+((!+ 

20+3+4+5+6+7+8+9+10+11+12) *26) ) 

Set 29 was created by subtracting the items 
already examined from set 2 the ADULT BASIC 
EDUCATION set. This (xooidB duplicate printing 
of any items. 



A second aspect of the search was begun to turn up industrial 
and fob training programs which dealt with basic skills. 



E-IT/INDUSTRIAL TRA 






S-E5 


30 


62 


E-E5 






S-E13 


31 


56 


S-E14 


32 


14 


S-E15 


33 


113 


S-E16 


34 


10 


S-E17 


35 


82 


S-E19 


36 


240 


C-30-36/+ 


37 


512 


C-37+16+17 


38 


591 


C-38*22 


39 


18 


P-99/5 




1-50 


P-29/5 




1-109 


P-39/5 




1-18 




IT /INDUSTRIAL TRAINING 

IT /INDUSTRIAL EDUCATION 

IT /INPLANT PROGRAMS 

IT /JOB TRAINING 

IT /OFF THE JOB TRAINING 

IT /ON THE JOB TRAINING 

IT /TRADE AND INDUSTRIAL EDUCATION 



INDUSTRIAL TRAINING 
was expanded and 
selected and then 
expanded to its thesaurus 
entry , This produced 
six additional relevant 
terms. 



30+31+32+33+34+35+36 



The seven industrial training terms 
were then combined by an OR 
operation to produce set Z7 , 



16+17+30+31+32+33+34+35+36 Set 37 was then added to sets 

16 and 17 which also relate to 
the same general concept, 

( 1+20+3+4+5+6+7+8+9+10+11+12) * ( 16+17+30+31+32+33+34+35+36) 

This sum was intersected with the basic education 
concept group (set 22) to get set 39, 



ITEMS HAVE BEEN PRINTED 
ITEMS HAVE BEEN PRINTED 
ITEMS HAVE BEEN PRINTED 



Finally three prints (P) were 
initiated of sets 99^ 29^ and 
39, Format 5 which contains the 
indexing^ cataloging y and 
abstracts for each document was 
chosen. After the print had 
been completed off-line at 
Lockheed the results were sent to\ 
the clearinghouse for forwarding 
to the evaluator. 
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SEARCH TITLES INSTRUCTIONAL DESIGN 
DATE; 06/27/69 

REQUESTOR; PERSON B 

SET 

COMMAND-OPERAND (S) NO. 



NO. IN 
SET 



DESCRIPTION OF SET 
(+-OR,*-AND,--NOT) 



« 


E-IT/ INSTRUCTIONAL 
S-El 


1 


1 




S-E3 


2 


113 


i 


S-E5 


3 


44 


1 


S-E7 


4 


141 


\ 

! 


S-E8 


5 


191 


1 . 

1 

1 

1 


E-E5 

S-Ell 


6 


56 


1 


S-E14 


7 


128 


! 

n 

& 

fS 


S-E15 


8 


15 


S-E16 


9 


43 


% 


E-IT/AUDIOVISUAL IN 
S-E5 


10 


127 




S-E6 


11 


1 




S-E7 


12 


1 




S-E2 


13 


365 




S-E9 


14 


34 


i, 

1 

i 


E-IT /EDUCATIONAL IN 
S-E3 


15 


134 


1 

i 

ji 


S-E5 


16 


107 


S-E9 


17 


1 


1 

\ 

i 


E-IT / INSTRUCTIONAL 
S-E5 


18 


998 


t ■< 


S-E8 


19 


68 




S-E6 


20 


46 


5 ■ 


C- 3+4+5+6+15+16 


21 


611 




C-l+2+7+8+9+lOfll+l 


22 


693 


1 1 

; i 

1 . 

\ 


C-22+17+18+19 


23 


1564 


{ 

R 

t 

i » 


C-21*23 


24 


140 


1 

1 


D-24 


m 




j 


K-24/2 







o 

ERIC 



K-24/3 

K-24/4 

K-24 



IT/INSTRUCTION AIDS 
IT/INSTRUCTIONAL AIDS 
IT/INSTRUCTIONAL DESIGN 
IT/INSTRUCTIONAL IMPROVEMENT 
IT/INSTRUCTIONAL INNOVATION 



IT/EDUCATIONAL STRATEGIES 
IT/INSTRUCTIONAL TECHNOLOGY 
IT /MULTIMEDIA INSTRUCTION 
IT/RESEARCH DESIGN 



IT /AUDIOVISUAL INSTRUCTION 
IT/AUDIOVISUAL INSTRUCTIONS 
IT/AUDIOVISUAL MATERIALS 
IT/AUDIOVISUAL AIDS 
IT/AUDIOVISUAL PROGRAMS 



IT/EDUCATIONAL IMPROVEMENT 
IT/EDUCATIONAL INNOVATION 
IT /EDUCATIONAL MATERIALS 



I 



* i 



1 



I 



IT/INSTRUCTIONAL MATERIALS 
IT/INSTRUCTIONAL MEDIA 
IT /INSTRUCTIONAL MATERIALS CENTERS 



3+4+5+6+15+16 

1+2+7+8+9+10+11+12+13+14 

17+18+19+1+2+7+8+9+10+11+12+13+14 

( 3+4+5+6+15+16) * ( 17+18+19+1+2+7+8+9+10+11+12+13+14) 
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I 

















E-IT/PROGRAMED INST 
S-E5 . 25 

S-E6 26 

S-E7 27 



540 

1 

202 



E-IT/PROGRAMED INST 
S-E8 28 

S-E9 29 



88 

8 



E-IT/COMPUTER ASSIS 

S-E4 

S-E5 

S-E6 

S-E7 



30 

31 

32 

33 



1 

105 

8 

68 



C-25-33/+ 

0-21*34 



34 

35 



761 



72 



D-35 



C-35-24 



36 41 



D-36 

K-36/2 

K-36/3 

K-36/4 

K-36/5 

K-36/9 

K-36/10 

K-36/13 

K-36/14 

K-36/16 

K-36/17 

K-36/18 

K-36/19 

K-36/21 

K-36/23 

K-36/24 

K-36/25 

K-36/28 

K-36/30 

K-36/31 

K-36/32 

K-36/33 

K-36/34 

K-36/36 

K-36/37 

K-36/38 

K-36/39 

K-36/40 

K-36/41 



IT /PROGRAMED INSTRUCTION 
IT /PROGRAMED MATERIAL 
IT /PROGRAMED MATERIALS 




IT /PROGRAMED TEXTS 
IT /PROGRAMED TUTORING 



IT/COMPUTED ASSISTED INSTRUCTION 
IT/COMPUTER ASSISTED INSTRUCTION 
IT/COMPUTER BASED LABORATORIES 
IT /COMPUTER ORIENTED PROGRAMS 



25+26+27+28+29+30+31+32+33 

( 3+4+5+6+15+16) * ( 25+26+27+28+29+30+31+32+33) 



POLISH STRING OVERFLOWS 2 LINES. NO ERROR. 
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i 



I 



I 



i 



1 

I 



1 



■I 






















H 



o 

ERIC 



D-99/1 

C-99*36 



E-IT/FILMS 

S-E5 

E-E5 

S-E12 

S-E14 

S-E16 

S-E17 

S-E27 



C-38-43 

C-38-43/+ 

C-21*45 



D-46 

K-46/1 

K-46/2 

K-46/3 

K-46/5 

K-46/6 

K-46/7 

K-46/8 

K-46/10 

K-46/11 

K-46 



P-99/5 



37 


28 


38 


187 


39 


1 


40 


183 


41 


6 


42 


30 


43 


5 


44 


185 


45 


339 


46 


27 



POLISH STRING OVERFLOWS 2 LINES. NO ERROR. 
IT/FILMS 



IT /MOTION PICTURES 
IT /INSTRUCTIONAL FILMS 
IT/SINGLE CONCEPT FILMS 
IT/SOUND FILMS 
IT/REPETITIVE FILM SHOWINGS 



38-43 

38+39+40f41+42+43 

( 3+4+5+6+15+16) *( 38+39+40f 41+42+43) 



1-181 ITEMS HAVE BEEN PRINTED. 
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SEARCH TITLE: 

DATE: 

REQUESTOR: 

SET 

COHMAND-OFERAND(S) NO. 



MULTIMEDIA TELEVISION 

06/04/69 

PERSON C 



NO. IN DESCRIPTION OF SET 

SET (+-OR,*-AND,— NOT) 



E-IT /MULTIMEDIA 

S-E6 

E-E6 

S-E12 

S-E13 

S-E14 

S-E15 

D-1 

K-1/2 

K-1/3 

K-1/4 

K-1/6 

K-1/7 

K-1/10 

K-1/14 



1 15 IT/MULTIMEDIA INSTRUCTION 

2 44 IT/INSTRUCTIONAL DESIGN 

3 191 IT /INSTRUCTIONAL INNOVATION 

4 998 IT/INSTRUCTIONAL MATERIALS 

5 128 IT/INSTRUCTIONAL TECHNOLOGY 



t. 1 


S-E4 


6 


155 


IT /TELEVISED INSTRUCTION 


■ 

1 


S-E5 


7 


88 


IT/TELEVISION 




S-E7 


8 


22 


IT /TELEVISION CURRICULUM 


f ^ 


S-E8 


9 


2 


IT/TELEVISION INSTRUCTION 


'i 

t' 

i 


D-9 

1 










E-IT/INSTRUCTIONAL 








1 


S-E5 


10 


153 


IT/ INSTRUCTIONAL TELEVISION 


1 


E-E5 








k 


S-E16 


11 


113 


IT/INSTRUCTIONAL AIDS 


IT 

s' 


E-IT/EDUCATIONAL TE 






i 1 


S-E5 


12 


238 


IT /EDUCATIONAL TELEVISION 


i 


E-E5 










S-E18 


13 


91 


IT /CLOSED CIRCUIT TELEVISION 




S-E23 


14 


26 


IT/OPEN CIRCUIT TELEVISION 


i 

I 


S-E24 


15 


13 


IT/TELECOURSES 


P- 

1 


C-6-15/+ 


16 


572 


6+7+8+9+10+11+12+13+14+15 


S 

1 


C-16-11 


17 


459 


(6+7+8+9+10+11+12+13+14+15) -11 


1 


^ C-11+4 

t 


18 


1083 


4+11 


k 


S-IT/EDUCATIONAL IN 19 


107 


IT/EDUCATIONAL INNOVATION 
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me 



liifliiiiili 






er|c 



C- 19+3+5 


20 


0-17*18*20 


21 


D-21 




K-21/1 




K-21/2 




K-21/4 




K-21/6 




K-21/7 




K-21/8 




\a~2U9 





E-IT/SIMULATION GAM 



23 

24 

25 

26 



S-E4 
E-E4 
S-EIO 
S-E12 

E-IT/GAMES 

S-E5 

E-E5 

S-Ell 

D-23 

D-24 

K-24/5 

K-24/8 

D-25 

K-25/2 

K-25/7 

K-25/10 

D-26 

K-26/1 

K-26/4 

K-26/5 

K-26/6 



E-IT /FILM LOOPS 
E-IT /FILMSTRIP 
E-IT /ROLE PLAYING 
S-E5 27 

D-27 
K-27/14 

P-99/5 

GOOD WORK THANK YOU 



403 3+5+19 

9 ((6+7+8+9+10+11+12+13+14+15)-11)*(4+11)*(3+5+19) 



22 103 IT/SIMULiVTION 



7 IT/TECHNIQUES 

8 IT /GAME THEORY 

25 IT/GAMES 

10 IT/EDUCATIONAL GAMES 



36 IT /ROLE PLAYING 



1-24 ITEMS HAVE BEEN PRINTED. 
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SEARCH TITLE: ASSESSMENT OF PERCEIVED ENVIRONMENT 

DATE: 06/25/69 

REQUESTOR: PERSON D 

SET 

COMMAND-OPERAND (S) NO. 



NO. IN 
SET 



DESCRIPTION OF SET 
(+-OR,*-AND,— NOT) 



I' 



I 



lERlC 



E-IT /COLLEGE AND UN 
S-E4 



S-E5 

S-E6 

S-E7 



1 

2 

3 

4 



C-1+2+3+4 



D-5 

K-5/1 

K-5/2 

K-5/3 

K-5/4 

K-5/5 

K-5/6 

K-5/7 



E-IT /COLLEGE CHARAC 
S-E4 6 

S-E5 7 

S-E6 8 



C-6+7+8 



D-9 

K-9/1 

K-9/2 

K-9/3 

K-9/4 



E-IT/ACTIVITIES IND 
S-E5 10 

S-E6 11 



C-10+11 



12 



D-12 

E-IT/ORGANIZATIONAL 
S-E3 13 

S-E6 1^ 

S-E7 15 

S-E8 16 



1 

4 

1 

1 



1 

2 

2 



3 

1 



1 

2 

1 

2 



IT/COLLEGE AND UNIVERSITY ENVIRONMENT SCALE 

IT/COLLEGE AND UNIVERSITY ENVIRONMENT SCALES 

IT/COLLEGE AND UNIVERSITY ENVIRONMENT SCALES %CUES< 

IT/COLLEGE AND UNIVERSITY ENVIRONMENTAL SCALES 



1+2+3+4 



IT/COLLEGE CHARACTERISTICS ANALYSIS %CCA< 
IT/COLLEGE CHARACTERISTICS INDEX 
IT/COLLEGE CHARACTERISTICS INDEX %CCI< 



6+7+8 



IT/ACTIVITIES INDEX 
IT /ACTIVITIES INDEX %AI< 



10+11 



IT /ORGANIZATIONAL BEHAVIOR DESCRIBER SURVEY 
IT/ORGANIZATIONAL CLIMATE DESCRIPTION QUESTIONNAIRE 
IT/ORGANIZATIONAL CLIMATE DESCRIPTIVE QUESTIONNAIRE 
IT/ORGANIZATIONAL CLIMATE INDEX 
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I 



i 



1 



I 



I 

i 

i 

I 



i 





















I 




C-13-16/+ 



17 



D-17 

K-17/1 

K-17/2 

K-17/3 

K-17/4 

K-17/5 

K-17/6 



e-it/high school CH 
S-E5 18 

D-18 
K-18/1 



E-IT/CIASSROOM CLIM 

S-E6 19 

S-E9 20 

D-19 

K^19/l 

K-19/2 

K-19/3 

K-19/4 



E-IT/ELEMENTARY SCH 

E-I /HIGH SCHOOL AT 

E-IT/COLLEGE ATMOSP 

E-IT/ENVIRONMENT 

S-E5 

S-E6 



E-E5 

S-E15 

S-E17 

S-E29 

S-E32 



E-IT/CLIMATE 

D-22 

K-22/1 

E-IT/PERCEIVED ENVI 
S-E6 27 

S-E7 28 



13+14+15+16 



IT/HIGH SCHOOL CHARACTERISTICS INDEX 



4 

103 



IT/CLASSROOM CLIMATE QUESTIONNAIRE 
IT/CLASSROOM ENVIRONMENT 



21 


24 


22 


1 


23 


63 


24 


35 


25 


136 


26 


38 



IT/ENVIRONMENT 

IT/ENVIRONMENTAL ASSESSMENT TECHNIQUE 



IT/EDUCATIONAL ENVIRONMENT 
IT /INSTITUTIONAL ENVIRONMENT 
IT/ENVIRONMENTAL INFLUENCES 
IT/ORGANIZATIONAL CLIMATE 



111 

36 



IT/PERCEPTION 
IT/PERCEPTION TESTS 
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I 



% 



t 



f 



i 



I 



E-IT /MEASUREMENT TE 
S-E2 


29 


1 


IT/MEASUREMENT INSTRUMENT INSTRUMENTS 


S-E3 


30 


167 


IT/MEASUREMENT INSTRUMENTS 


S-E5 


31 


128 


IT/MEASUREMENT TECHNIQUES 


S-E6 


32 


1 


IT /MEASURING INSTRUMENTS 


E-E5 

S-E12 


33. 


34 


IT/INSTRUMENTATION 


S-E22 


34 


54 


IT /MEASUREMENT 


S-E25 


35 


208 


IT /TESTING 


E-E25 

S-Ell 


36 


1 


IT/TESTING METHODS 


S-E33 


37 


46 


IT/PREDICTIVE VALIDITY 


S-E39 


38 


151 


IT/TEST CONSTRUCTION 


S-E43 


39 


178 


IT /TESTS 


C-2(M-21+23+24+25+26 


40 


366 


20+21+23+24+25+26 


C-27-30/+ 


41 


307 


27+28+29+30 


C-31-39/+ 


42 


685 


31+32+33+34+35+36+37+38+39 


C-41+42 


43 


913 


27+28+29+30+31+32+33+34+35+36+37+38+39 


0-40*43 


44 


36 


( 20+21+23+24+25+26) * ( 27+28+29+30+31+32+33+34+35+36+37+38+39) 



i 



t 



D-44 

K-44/2 

K-44/3 

K-44/8 

K-44/9 

K-44/10 

K-44/14 

K-44/20 

K-44/22 

K-44/26 

K-44/28 

K-44/30 

K-44/31 

K-44/33 



1 



I 



■I 



E-IT/COPING ANALYST 
E-IT/CUES 



S-E5 

D-45 

E-IT/CCI 

S-E6 

D-46 

E-IT/OCDQ 

E-IT/HSCI 



45 



46 



IT/CUES 

IT/CCQ 
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Ittsus&m 













E-IT/ECOLOGY 



S-E4 


47 


14 


S-E5 


48 


30 


C-47+48 


49 


43 



IT/ECOLOGICAL FACTORS 
IT/ECOLOGY 



43 47+48 



D-49 

K-49/1 

K-49/4 

K-49/9 

K-49/10 

K-49/12 

K-49/21 

VL-U9/22 

K-49/36 

K-49/37 



P-99/5 



1-45 ITEMS HAVE BEEN PRINTED 
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SEARCH TITLE: 
DATE: 

REQUESTOR: 



COMPUTERS 
05/27/69 
PERSON E 

SET NO. IN 



COMMAND-OPERAND (S) NO. SET 



DESCRIPTION OF SET 
(+-OR**-AND,— NOT) 






t 






% 






I 




E-IT /COMPUTERS 

S-E3 

S-E2 

S-E5 



1 

2 

3 



1 

4 

130 



IT /COMPUTER SCIENCE EDUCATION 
IT/COMPUTER SCIENCE 
IT/COMPUTERS 



D-1 

D-2 



E-IT/COMPUTERS 

E-E5 

E— E20 

E-E17 

E-E17 

E-E15 

S-E13 

E-IT/TRANSFORMATION 



S-E3 



C-4*5 



54 

45 

5 



IT /LINGUISTIC THEORY 
IT /TRANSFORMATION GENERATIVE GRAMMAR 



4*5 



D-6 

D-6 

K-6/1 

K-6 



E-AU/DYEN 

S-E6 

D-7 

K-7/5 

K-7/6 



AU/DYEN, ISIDORE 



E-AU/HAJDU, PETER 

S-E5 

D-8 

K-8 



8 



AU/HAJDU, PETER 



S-AU/CHOMSKY, NOAM 

D^9 

K-9 



AU/CHOMSKY, NOAM 



73 






I 






J 



’1 



I 









1 






..^1 

■S 



A 



4 



i 



I 



I 







E-IT/THESAURI 

S-E6 


10 


1 


IT/THESAURUS OF ENGINEERING AND SCIENTIFIC TERMS 


D-10 

E-IT/ENGINEERING 

S-E5 


11 


50 


IT/ENGINEERING 


S-IT/THESAURI 


12 


23 


IT/THESAURI 


C-ll*12 


13 


1 


11*12 


D-13 

E-IT/EURATOM 

E-IT/ALGOL 

S-E5 


14 


1 


TI/ALGOL 


D-14 

E-IT/SNOBOL 

S-E5 


15 


2 


IT/SNOBOL 


D-15 

K-15/1 

D-15/2/2 

K-15/2 

E-IT/COMPUTER ASSIS 
S-E4 


16 


1 


IT /COMPUTED ASSISTED INSTRUCTION 


S-E5 


17 


105 


IT/COMPUTER ASSISTED INSTRUCTION 


S-E7 


18 


68 


IT /COMPUTER ORIENTED PROGRAMS 


E-IT/INFORMATION PR 
S-E5 


19 


104 


IT /INFORMATION PROCESSING 


S-E6 


20 


119 


IT /INFORMATION RETRIEVAL 


S-E7 


21 


17 


IT /INFORMATION SCIENCE 


E-IT/CYBERNETICS 

S-E5 


22 


23 


IT/CYBERNETICS 


E-IT /PROGRAMMING LA 
C-16-22/+ 


23 


357 


16+17+18+19+20+21+22 


C-1-10/+ 


24 


233 


1+2+3+4+5+ ( 4*5) +7+8+9+10 


C-11-22/+ 


25 


404 


ll+12+(ll*12)+14+15+16+17+18+19+20+21+22 


C-24-25/+ 


26 


572 


1+2+^4+5+(4*5)+7+8+9+10+11+12+(11*12)+14+15+16+17+18+ 


S-IT/COMPUTATIONAL 


27 


37 


19+20+21+22 

IT/COMPUTATIONAL LINGUISTICS 


K-27 

E-IT/LINGUISTICS 

S-E5 


28 


243 


IT /LINGUISTICS 


E-IT/LANGUAGES 

S-E5 


29 


122 


IT/LANGUAGES 


E-IT/STATISTICS 

S-E5 


30 


13 


IT/STATISTICS 
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1 C-28+29 

1 


31 


348 


28+29 


1 C-30*31 

1 




C 




It 

|! S-E2 


32 


379 


IT/STATISTICAL DATA 


1 E-E2 








1 E-E2 








1 S-Ell 


33 


35 


IT/COMPARATIVE STATISTICS 


1 S-E25 

K 


34 


256 


IT/STATISTICAL ANALYSIS 


1 C-32-34/+ 


35 


644 


32+33+34 


1 C-35*31 

1 


36 


11 


(28+29) *(32+33+34) 


( D-36 

j _ ; 








[ P-99/2 








j P-23/2 




1-357 


ITEMS HAVE BEEN PRINTED. 


i 

1 [ P-23/4 




1-357 


ITEMS HAVE BEEN PRINTED. 



i' 



r 



t: 



I 
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1 



SEARCH TITLE: MICROFICHE READER/PRINTERS 

DATE: 06/13/69 

REQUESTOR: PERSON F 



I 






SET NO. IN DESCRIPTION OF SET 

COMMAND-OPERAND (S) NO. SET (+«OR,**AND,-"NOT) 



E-IT /MICROFICHE 

E-E5 

E-E6 

S-EIO 

D-1 

K-1/3 

D-1/2/2 

D-1/2/4-7 

K-1 



7 IT/MICROFICHE 



E-IT /TECHNOLOGY 

S-E4 

D-2 

e-it/data control 

E-IT /MICROPHOTOGRAP 

E-E3 

S-E3 

D-3 



11 IT/TECHNOLOGICAL DEVELOPMENT PROJECT 



7 IT/MICROFILM 



E-AU /HOLMES 
E-AU /COOMBS 



E-TI /GUIDE 
E-TI /GUIDE L 
E-TI /GUIDE P 



E-PS /EDUCATIONAL FA 
S-E6 ^ 

D-4 



20 PS /EDUCATIONAL FACILITIES LABS. INC., NEW YORK, N.Y. 



C-3*4 



1 3*4 






D-5 

E-PS /FORD FOUNDATIO 
S-E6 6 

D-6 



6 PS/FORD FOUNDATION, NEW YORK, N.Y. 



J 






E-IT/FACSIMILE 

S-E7 7 

D-7 

E-IT /TELECOMMUNICAT 
S-E8 8 

D-8 

E-AU/PIERCE 

S-E9 9 

D-9 

E-AU/PIERCE, J.R. 



2 IT/FACSIMILE TRANSMISSION 

1 IT/TELEPHONE COMMUNICATIONS INDUSTRY 

1 AU/PIERCE, J.L. 



E-PS /BELL TELEPHONE 
S-E6 10 

D-10 



1 PS /BELL TELEPHONE LABS., INC. MURRAY HILL, N.J. 




E-IT/ SATELLITE 

E-IT/COMSAT 

S-E5 11 



3 IT /COMSAT 



D-11 

D-11/5/1-3 

E-PS /HARVARD UNIVER 
S-EA 12 

D-12 

E-IT /ARIZONA 
S-IT/HARVARD UNIVER13 
S-IT /COLUMBIA UNIVEIA 
S-IT /UNIVERSITY OF 15 
C-13*1A*15 

E-IT/LIBRARY AUTOMA 

S-E2 16 

S-E3 17 

D-16 

D-17 

K-17/1 



3 PS /HARVARD UNIV., CAMBRIDGE, MASS., LAB. RES. INS 



A IT/HARVARD UNIVERSITY 

5 IT/COLUMBIA UNIVERSITY 

A IT/UNIVERSITY OF CHICAGO 

0 



10 IT/LIBRARY ACQUISITION 

1 IT /LIBRARY ACQUISTION 



E-AU/RUECKING 
E-AU/DILLON, H.W. 
E-AU/OVERHAGE, C.F. 





E-PS/INTREX 
E-PS /MIT PROJECT IN 
E-PS /PROJECT INTREX 
E-PS /MASSACHUSETTS 
S-EA 18 

D-18 

E-PS /ELECTRONIC SYS 



12 PS /MASSACHUSETTS INST. OF TECH., CAMBRIDGE 
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|i 



E-AU/STEVENS 
E-AU/BENENFIELD, AL 
E-AU/SIMONTON 



E-IT/CRT 

E-IT/VISUAL DISPLAY 
E-IT/CATHODE RAY TU 
E-IT/TERMINAL 



SEARCH TITLE: SOFT COPY 



DATE: 

REQUESTOR: 



06/13/69 
PERSON F - 2 



SET NO. IN 

COMMAND-OPERAND (S) NO. SET 



DESCRIPTION OF SET 
(+-OR,*-AND,— NOT) 












I 



^ t 






! 



I f 



h 



er|c 



E-IT/SOFTPCOPY 

E-IT/PHOTODUPLICATI 



E-PS /IBM 

E-PS / INTERNATIONAL 

S-E4 

S-E6 

S-E7 



1 

2 

3 



1 

8 

1 



PS/INTERNATIONAL BUSINESS MACHINE CORP. 

PS /INTERNATIONAL BUSINESS MACHINES CORP. 

PS /INTERNATIONAL BUSINESS MACHINES CORP., ONEGO, N 



C-1-3/+ 



S-IT/LIBRARIES 



10 

82 



1+2+3 



IT/LIBRARIES 



C-4-5/+ 



6 



C-4-5/* 



92 

0 



1+2+3+5 



E-AU/RABINOW 

S-E6 7 

D-7 

E-IT/OPTICAL CHARAC 
S-E6 8 

S-E7 9 



AU/RABINOW, BARNEY 



1 IT/OPTICAL CHARACTER RECOGNITION %OCR< 

1 IT/OPTICAL SCANNERS 
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mm 






I 



I 



s 



'4 





















C-8-9/* 



10 1 8*9 



D-10 

E-PS /CONTROL DATA C 
S-E2 11 

D-11 

E-IT/LIBRARY TECHNI 
E- IT/TELETYPEWRITER 
S-E4 12 

S-E3 13 



1 PS /CONSORTIUM OF UNIVERSITIES OF METRO. WASHINGTON 



2 IT/TELETYPEWRITER 

1 IT/TELETYPE 



C-12-13/+ 



14 3 12+13 



D-14 

E-IT/ULTRAFICHE 

S-E6 



D-15 

E-AU/GREENBERG 



15 



20 IT/UNCOMMONLY TAUGHT LANGUAGES 








k 



i 



i 





i 




SEARCH TITLE! LITERATURE ON LECTURING 
DATE: 06/11/69 

REQUESTOR: PERSON G 

SET NO. IN DESCRIPTION OF SET 

COMMAND-OPERAND (S) NO. SET (+-OR,*«AND,--NOT) 



S-IT/LECTURE 



1 35 IT/LECTURE 



D-1 

K-1/3 

K-1/4 

K-1/5 

K-1/6 

K-1/11 

K-1/12 

K-1/15 

K-1/16 

K-1/17 

K-1/18 

K-1/23 

K-1/29 

K-1/30 

K-1/31 

K-1/34 

K-1/35 



E-IT /PRESENTATION 
E-IT/LECTURE 



E-E5 

S-Ell 2 
S-E14 3 
E-Ell 

S-E16 4 
S-Ell 5 
S-E18 6 
S-E22 7 
S-E20 8 



432 

96 

1 

61 

9 

45 

99 



IT/TEACHING TECHNIQUES 
IT/SPEECHES 

IT /LEAD LECTURE PLAN 
IT/CLASSROOM TECHNIQUES 
IT/MASS INSTRUCTION 
IT/SMALL GROUP INSTRUCTION 
IT /REINFORCEMENT 



C-5*8 



0 



C-2*8 



9 7 2*8 
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I 




I 

I 



% 



I 



I 

.'t- 



■| 

I 

I 

4 






i 








'• »r» 






I 



I 



I 




I 

I 

h' 

I 




I 

I 

S' 



i 



/ 











I 



D-9 

K-9/2 

K-9/3 

K-9/4 

D-9/4/6 

K-9/6 

D-6 

K-6/5 

D-4 

K-4/1 



C-3*8 

D-10 

K-10/2 


10 


2 


C-7*8 

D-11 

K-11/1 


11 


1 


E-IT/TEACHING TECHN 




S-E2 

K-12 


12 


2 


S-E3 

K-13 


13 


15 


S-E4 

K-14 


14 


17 


C-8*13 


15 


1 


C-8*14 

D-15 

D-16 


16 


1 


S-IT/TEACHING METH017 


563 


C-17*8 

D-18 

K-18/5 


18 


6 



-IT/MOTIVATIONAL T 
S-IT/MOTIVATION TEC19 47 

C-19*2 20 5 

D-20 

K-20/3 

K-20/4 

P-99/5 1-59 



3*8 

7*8 

IT/TEACHING SITUATION REACTION TEST 
IT/TEACHING SKILLS 
IT/TEACHING STYLES 

8*13 

8*14 

IT /TEACHING METHODS 
8*17 

IT/MOTIVATION TECHNIQUES 
2*19 

ITEMS HAVE BEEN PRINTED. 
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SEARCH TITLE: PROGRAMED INSTRUCTION IN READING 

DATE: 07/23/69 



REQUESTOR: 



PERSON H - 1 
SET NO. IN 



E-IT /PROGRAMED INST 

S-E5 

S-E7 

S-E9 

E-E5 

S-E23 

S-E31 

C-1+2+3+4+5 

E-IT /READING 

S-E5 

S-E6 

S-E7 

S-E8 

E-E5 

S-E12 

S-E16 

S-E22 

S-E23 

S-E24 

S-E25 

S-E32 

S-E36 

S-E41 

S-E53 

S-E54 

S-E63 

S-E69 

E-E25 

S-Ell 

S-E14 

S-E15 



). 


SET 


1 


540 


2 


202 


3 


8 


4 


15 


5 


48 


6 


651 


7 


148 


8 


53 


9 


174 


10 


1 


11 


102 


12 


24 


13 


1 


14 


26 


15 


8 


16 


13 


17 


58 


18 


172 


19 


34 


20 


59 


21 


13 


22 


63 


23 


27 


24 


1 


25 


147 


26 


563 



description OF SET 
(+-OR,*-AND,— NOT) 



IT /PROGRAMED INSTRUCTION 
IT/PROGRAMED MATERIALS 
IT /PROGRAMED TUTORING 



IT /MULTIMEDIA INSTRUCTION 
IT/RESPONSE MODE 

1+2+3+4+5 



IT/READING 

IT/READING ABILITY 

IT/READING ACHIEVElffiNT 

IT /READING AND STUDY SKILLS LABORATORY 



IT /BEGINNING READING 
IT/DEVELOPMENTAL READING 
IT /GROUP READING 
IT /INDEPENDENT READING 
IT /INDIVIDUAL READING 
IT /INDIVIDUALIZED READING 
IT/REMEDIAL READING 
IT /LANGUAGE ARTS 
IT/INITIAL TEACHING ALPHABET 
IT/READING DIFFICULTY 
IT /READING FAILURE 
IT/READING RE^INESS 
IT /RETARDED READERS 



IT /INDIVIDUALIZED INSTRUCTION 
IT /READING PROGRAMS 
IT/TEACHING METHODS 



i . 
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C-6+ 24+26 


27 


1167 


1+2+3+4+5+24+26 


C-7-17/+ 


28 


508 


7+8+9+10fll+12+13+14+15+16+17 


C-18-23/+ 


29 


340 


18+19+20+21+22+23 


C-25+28+29 


30 


776 


25+7+8+9+10+11+12+13+14+15+16+17+18+19+20+21+22+23 


C-27*30 


31 


105 


( 1+2+3+4+5+24+26) * ( 25+7+8+9+10+11+12+13+14+15+16+17+ 

18+19+20f21+22+23) 


D-31 


P-31/5 




1-105 


ITEMS HAVE BEEN PRINTED. 



VERY INTERESTING 






INDIVIDUALIZED INSTRUCTION 



SEARCH TITLE: 8MM SINGLE CONCEPT FIL!fS 

DATE: 07/22/69 

REQUESTOR: PERSON I - 1 

SET NO. IN DESCRIPTION OF SET 
COMMAND-OPERAND (S) NO. SET (+«OR,**AND,—NOT) 



E-IT/SINGLE CONCEPT 






S-E5 


1 


6 


E-E5 






S-Ell 


2 


365 


S-E14 


3 


183 


S-EIO 


4 


187 


E-EIO 






S-E12 


5 


1 


S-E22 


6 


28 


S-E27 


7 


5 


C-1-7/+ 


8 


616 


E-IT/INDIVIDUALIZED 






S-E5 


9 


1 


S-E4 


10 


22 


S-E6 


11 


61 


S-E9 


12 


2 


E-E4 






S-E12 


13 


229 


E-E13 






C-9-13/+ 


14 


291 


C-8*14 


15 


75 


K-15/1 






K-15/4 






K-15/6 






K-15/7 






K/15/12 






K-15/13 






S-IT /DIESEL ENGINES 


16 


57 


C-15-16 


17 


21 



IT /SINGLE CONCEPT FILMS 

IT/AUDIOVISUAL AIDS 
IT/INSTRUCTIONAL FILMS 
IT/FILMS 



IT/MOTION PICTURES 
IT /FILM PRODUCTION 
IT /REPETITIVE FILM SHOWINGS 

l+2+3+A+5+6f7 



IT /INDIVIDUALIZED INSTRUCTION 
IT/INDIVIDUALIZED CURRICULUM 
IT/INDIVIDUALIZED PROGRAMS 

IT/INDIVIDUALLY PRESCRIBED INSTRUCTION %IPI< 
IT /INDIVIDUAL INSTRUCTION 



9+10+11+12+13 

(1+2+3+4+5+6+7)* (9+10fll+12+13) 



IT /DIESEL ENGINES 

( ( 1+2+3+4+5+6+7 ) * ( 9+10+11+12+13) ) -16 
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D-16 

D-16/2/55-57 

D-1 

K-1/1 

K-1/2 

K-1/3 

K-1/4 

K-1/5 

K-1/6 

D-12 

K-12/1 

K-12/2 



E-IT/8MM 

p_99/5 1-14 ITEMS HAVE BEEN PRINTED. 

VERY GOOD. I FOUND THIS SERVICE VERY INTERESTING, INFORMATIVE, AND HELPFUL 
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SEARCH TITLE: 
DATE: 

REQUESTOR: 



I 



i 



I; 



i 



I 






I 
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MULTI-MEDIA INSTRUCTION 

07/22/69 

PERSON 1-2 



SET NO. IN 

COMMAND-OPERAND (S) NO. SET 



DESCRIPTION OF SET 
(+-OR,*-AND,— NOT) 



E-IT/MULTI-MEDIA IN 

E-IT/MULTIMEDIA INS 

S-E5 

E-E5 

S-E13 

S-EX5 

E-E13 

S-Eli 



2 

3 



E-IT/READING 

S-E8 

S-E5 

E-E5 

S-Ell 

S-E14 

S-E15 

S-E16 

S-E22 

S-E25 



5 

6 



7 

8 
9 

10 

11 

12 



C-2-4/+ 

C-5-12/+ 



13 



14 



C-5+6+7+8+9+1CH-11+1 15 
C-5-12/+ 16 
C-13*14 17 



D-17 

K-17/1 

K-17/3 

K-17/4 

K-17/5 

K-17/6 

K-17/7 

K-17/8 

K-17/9 



15 IT /MULTIMEDIA INSTRUCTION 



191 

128 



1 

148 



59 

22 

10 

24 

1 

13 



403 

26 

264 

264 

9 



IT /INSTRUCTIONAL INNOVATION 
IT /INSTRUCTIONAL TECHNOLOGY 



4 107 IT /EDUCATIONAL INNOVATION 



IT/READING AND STUDY SKILLS LABORATORY 
IT/READING 



IT/BASIC READING 
IT/CONTENT READING 
IT/CREATIVE READING 
IT /DEVELOPMENTAL READING 
IT/GROUP READING 
IT/INDIVIDUALIZED READING 



2+3+4 



5+6+7+8+9+10+11+12 



5+6+7+8+9+10+11+12 



5+6+7+8+9+10+11+12 
( 2+3+4) * ( 5+6+7+8+9+10+11+12) 
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D-1 

K-1/1 

K-1/5 

K-1/6 

K-1/8 

K-1/9 

K-1/11 

K-1/12 

K-1/14 

K-1/15 

P-99/5 



1-17 ITEMS HAVE BEEN PRINTED. 



VERY GOOD. THANK YOU FOR YOUR ASSISTANCE. 



I 



l 
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